bc_thesis/en/chap05.tex

\chapter{Evaluation}\label{ch:evaluation}

To show that Birdvisu is a usable project, we have tried to use it in several
network systems, both synthetic and already deployed. Here we summarise our
experience with using it.

Because our project is mainly an interactive program, we do not present any
performance or other measurements. Therefore, we rather focus on the subjective
feeling of the project and evaluate the design decision from chapter~\ref{ch:design}.

\section{Gennet}

Naturally, Gennet, as described in section~\ref{s:gennet} is supported quite
well. Since we had complete control of the whole network, we could test e.g.
multi-area OSPF deployment, which is not common in real network systems, as far
as we are aware.

However, only small networks can be simulated in Gennet, because each router
requires disk and memory, which may be quite scarce.

\section{Our home network}

At home, the author relies on a dynamic routing to switch between falling
uplinks and to provide a simple way to address virtual machines across the
network system. This was one of the reasons we decided to look for
visualisations of OSPF topology state.

Since it only spans a single room, this system is even smaller than Gennet,
containing only three routers and less than 10 networks (the exact number
depends on which devices are up at the time). We believe that major issues do
not show up at this scale.

Naturally, Gennet can be connected to the home network. At that point, our
approach to laying out vertices starts feeling suboptimal, because edges cross
unnecessarily often. The connections are clear, however, and this can be
alleviated by using a fixed layout in a file. In the future, this could be
addressed by using a force-based approach for the automatic layout.

\section{Department of Applied Mathematics}

The department which advised this thesis also uses OSPF in its infrastructure.
Again, the main purpose is to address containers and virtual machines in the
system. The topology consists of 5 routers and about 27 networks, most of which
are stub.

As in the previous case, the main issue is the automatic vertex layout. Also, since most of the
networks are stub (and often only contain a single host), the graph could
position these networks near each other, or even collapse them into one vertex.
Birdvisu does not unfortunately support this at the moment.

\section{Czela.net}

Czela.net~\cite{czelanet} is a network system run by a community of network
enthusiasts in Čelákovice and the surrounding area. There are about 1600 people
connected to Czela.net. This is the typical OSPF
deployment, whose main task is to dynamically provide fallbacks in case of
outages. It is also an example of a larger network, with 45 routers, 32 transit
networks and 178 external routes\footnote{Most of these would be stub routes in
other deployments, but there is little difference from the topology perspective.}.

Unfortunately, their infrastructure does not use BIRD anywhere and we were not
able to connect our instance to any of their transit networks. Therefore we
only tested displaying the topology based on a dump from one of their MikroTik
routers, which we manually converted to mimic BIRD's format. We believe this
should not affect the performance much, because the retrieval of data from BIRD
is only occasional and once BIRD dumps the topology, the procedure is the same
as with loading the topology from file.

This turned out to be helpful for both us and Czela.net. We discovered that our
method of highlighting of costs does not work well with too big range of the
costs, and on the other hand, we found several misconfigurations in their
network (some routers had external routes to transit networks, a few networks
with routing-unaware clients were used as transit), even without any other
knowledge of their system.

We did not notice any performance issues when dealing with the topology.

\bigskip

Unfortunately, we did not have the opportunity to test in a large network (the
Czela.net's one is the largest we tested), so we do not really know the limits
of Birdvisu. Those are even hard to guess, because while we are trying to
use rather fast algorithms, they are implemented in Python, which can sometimes
be quite slow, and more importantly, does not support threads well. The heavy
use of hash tables and indirection can also impair performance.

Overall, the testing did not discover any important problems with the design of
Birdvisu, we are only aware of issues related to small parts of the project.