Deep Learning Blog | Bitfusion

Bitfusion Profiler: A Quick Introduction

Written by maciej | Oct 22, 2015 3:15:58 PM

Last week we released the Bitfusion Profiler as a first product out of Bitfusion Labs. Profiler enables one to quickly evaluate application performance across a variety of hardware and software configurations. It automatically detects limitations and whether a particular application might benefit from larger memory footprints, multiple sockets, more cores, larger disk drives or even a different cloud provider. Then Profiler suggests optimal configurations, helping users determine which instance types are fastest and which offer the best value. We built Profiler initially to meet our internal needs, and now because of customer demand, have put it out for everyone to use. Below is a short overview of how one can get started with the Profiler including an overview of the functionality and a quick tutorial of how one can get started quickly by modifying a sample application workload.

Getting Started

There are three easy steps one can follow to get up and running with the Profiler: First, create your application in the browser using the online terminal. Second, let the Profiler run after submitting the application. Third, analyze the generated reports to obtain insights about the workload.

The figure above shows the main dashboard for the Profiler which consists of two sections: My Applications and Public Applications. The first time when you log in, the My Applications section will be empty as you have not created any personal applications yet. The Public Applications section on the other hand has various applications benchmarks which we have created so that users can get started quickly. If you are curious about a particular application you can click the name of the application which will bring up a window showing all available notes about that particular application as shown below.

You can click the “Copy” button for any of the public workloads which will create a copy of the public workload in your personal My Applications section. To follow along with this tutorial please copy the “Blender” application workload - this may take a couple minutes so please be patient. In the meantime, you can click any of the “View Report” buttons in the Public Applications section. This will bring up a Profiler generated report for these applications from a previous run, as show in the Figure below.

The top chart shows the run-time of the application vs. the machines (instances) on which the applications was executed. For this chart a shorter bar indicates better performance, as this indicates a shorter run-time. The bottom chart shows how many application runs can be executed for every dollar spent with the cloud provider. In this case a longer bar is better since we want to perform as many runs for every single dollar we spend. In short, you want to use the top chart if you are interested in raw performance, and use the bottom chart to find the most cost efficient machine. On the left-hand side of the report you can find several filters which you can apply to the charts to customize the report. For example, you can narrow down your search by the instance type should there be a particular set of machines you are specifically interested in. On the top right you can find the “Share Report” button which will give you a link you can utilize to share the report with other people who you think may find it interesting.

Any of the bar charts are clickable and yield detailed application performance information for each instance. The figure above shows detailed performance for for the AWS c4.2xlarge, including CPU utilization for each of the cores, memory utilization, and disk usage. You can zoom-in on each data-point by selecting an area of the graph, and you can toggle the visibility of each visible data set by clicking on it. For example, you can deselect all CPU cores except for one to only see the utilization of that specific core.

Clicking on the Dashboard button will return you to the dashboard screen where the workload you previously copied should now be in the Ready state as shown in the figure above. At this point have various options in the Actions column: Either you can click the “Run” button which will start a new profiling run for this application, or if you want to make a few changes you can click the “Edit” button to make modification to the existing application.

Editing and Profiling a Copied Application

After you click the “Edit” button you will be taken to the Application screen as shown below. This is the exact screen you would see if you clicked the “Create New Application” button on the Dashboard, except in that case you would get a completely clean Linux environment without any installed application - since we are lazy and don’t want to start completely from scratch we chose instead to edit a copied application.

The application screen consists of several sections:

Linux Terminal - This terminal is a complete Linux terminal which serves as your main point of interaction when modifying or building your application workload.

Command Box - The command box contains the command or script which you want to profile. We’ll take a look at the contents of this script a little later. Below the command box you will find the “Test Command” button which is used to run the command in the command box to ensure everything runs as expected. You should always run the Test Command before pressing the Finish button to make sure there are no issues during the profiling run.

Notes Section - In the notes box you can record anything you may find useful in the future. In this case you will see that we wrote something about the “Blender” application and then went on the provide detailed instructions on how the application was built from scratch - in case someone would like to re-create it. Also note that anything you put in the notes section is what is shown when the application name is clicked on the Dashboard page.

Action Buttons - Save, Finish, Cancel

Save Button: When clicked a snapshot is taken of your progress and then you will be returned to the application screen so that you can continue working. Feel free to save often so that you can resume working from a most recent version should your browser crash or should you get disconnected from the internet.

Finish Button: When clicked there are two possible outcomes. If no command was specified in the command box, then the application state will be saved and you will be returned to the Dashboard. If a command was specified in the command box, then once the application is saved it will be automatically submitted for profiling.

Cancel Button: When clicked, all changes you have made since the last save will be discarded and you will be returned to the Dashboard. Notice, that if you have never saved an application, then the application will be completely deleted from the Dashboard as well. This can only happen for applications which are being created from scratch - for copied applications the act of copying created an automatic save point.

To modify the existing Blender application, click on the Linux terminal and then execute the following command:

vi /root/run.sh

Which will show you the following file contents:

#!/bin/bash
blender -noaudio -b /root/BMW27GE.blend -f 0 > run.log

As you can see, we are executing the Blender command in bash mode for just a single time. This command runs relative fast, and for real benchmarking it rarely makes sense to run a short running command just a single time, so let’s change it to the following to run a few more iterations:

#!/bin/bash
blender -noaudio -b /root/BMW27GE.blend -f 0 > run1.log
blender -noaudio -b /root/BMW27GE.blend -f 0 > run2.log
blender -noaudio -b /root/BMW27GE.blend -f 0 > run3.log

If you are not familiar with Vi you may want to briefly review the following tutorial.

Now that the run.sh file is updated let’s test it by pressing the “Test Command” button. During the test run the Terminal is disabled and you will see the screen below. You can stop the test run at any time using the “Stop Test” button.

Now what the test run is complete press the Finish button. This will take you back to the Dashboard screen and you will see that the application is being saved as indicated by the status column. Once the saving competes the application will be submitted automatically for profiling. The saving and profiling take a few minutes - you will receive an email once the profiling run is complete. Alternatively, you can follow the progress of the run via the progress bar which becomes visible on the Dashboard once the profiling run starts as shown in the figure below. Once you receive the email you can click the enclosed link to view the results, or you can return to the dashboard where you will see a “View Report” button next to the application.

Another feature that the Profiler offers is the ability to compare various reports. This can be done quickly by selecting multiple applications which have reports. For example, select the application which you just profiled by using the check-box to the left, and also select the original Blender application in the Public Applications sections and then click the “Compare Reports” button at the top of the page, as shown in the figure below.

This will bring up the report similar to the one shown below, where the bars in blue represent the original Blender application, while the orange bars represent the application you just modified. As you can see from the numbers, the run-times for the new application are approximately three times longer than previously - this matches our expectations as we ran the command three times instead of once.

Congratulations, you have successfully modified an application, profiled it, and compared it against a previous application run.

Any questions or feature requests? Please post them in the comments below or contact us directly.