Download exacqVision Virtual Appliance User Manual www.exacq.com

exacqVision Virtual Appliance User Manual
Contents
1 Introduction ......................................................................................................................................................... 2
2 Requirements ....................................................................................................................................................... 2
3 Additional exacqVision Virtual Servers ................................................................................................................ 3
4 Installation ........................................................................................................................................................... 3
5 exacqVision Web Service ..................................................................................................................................... 5
6 Upgrading Hardware ............................................................................................................................................ 6
7 Configuration ....................................................................................................................................................... 8
8 Troubleshooting and Monitoring ....................................................................................................................... 10
9 Configuring Virtual Disk Storage ........................................................................................................................ 12
10 Configuring Additional Networking ................................................................................................................. 31
11 Performance Monitoring ................................................................................................................................. 34
12 Upgrading exacqVision to a New Release........................................................................................................ 38
13 Pre-Migration Data Sheet ................................................................................................................................ 43
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 1 of 44
4/12/2013
1 Introduction
The exacqVision Virtual Appliance server contains a Linux-based installation of exacqVision server in a single, compressed,
transportable file that can easily be deployed on a VMware host server in minutes. The server runs unlicensed with one camera, and
additional cameras can be added with an Enterprise license.
This document is written for a VMware administrator installing the exacqVision virtual appliance in an existing VMware cluster. It is
assumed that the VMware host has been properly installed and configured with host hardware certified by VMware. Hardware can
be validated at the VMware Compatibility Guide website.
2 Requirements
Before deployment, system administrators should calculate video storage space and networking requirements using the
configuration calculator found at: https://exacq.com/config/.
Minimum Provisions







VMware vSphere 4i or 5i ESX Hypervisor
2 virtual cores with a reservation of 1000MHz
2GB virtual RAM
140GB of disk space (100GB allocated to video storage)
Disk I/O capacity of 125% the storage data rate calculated at https://exacq.com/config/.
Sufficient network bandwidth to support incoming video streams
One EVES-01 exacqVision Enterprise Server license
Recommended Provisions





4 virtual cores with a reservation of 6000MHz
6GB of virtual RAM
Additional video storage space to meet video storage requirements
Disk I/O capacity of 150% the storage data rate calculated here: https://exacq.com/config/
Two independent subnets for cameras and exacqVision Client connections (four maximum)
The exacqVision Server can address up to 64 virtual disks with a limit of 2TB each. The server supports Logical Volume Management
(LVMs) using virtual disks, which allows presentation of volumes greater than 2TB to the system. See section 9, “Configuring Virtual
Disk Storage,” for more information about managing virtual storage on the appliance.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 2 of 44
4/12/2013
3 Additional exacqVision Virtual Servers
You can install multiple exacqVision Virtual appliances on the same host of the same vSphere cluster using unique guest names.
Each exacqVision Virtual Appliance must have a separate exacqVision Server license. Camera licenses are generated based on the
MAC address of each guest and cannot be combined or duplicated, even if both exacqVision Virtual servers reside on the same host.
Each exacqVision Virtual Appliance comes with the exacqVision Client and the exacqVision Web Service installed. Because of limited
video display capabilities of virtualized appliances, exacqVision client is recommended only for diagnostic activities when used with
exacqVision Virtual appliances (that is, it is not recommended for live video display).
4 Installation
The exacqVision Virtual server can be downloaded from the Exacq Technologies Website or through the VA Marketplace accessible
from the File menu in the vSphere Client. The downloaded file can be saved to any computer that has the vSphere client installed
and which has an account with permission to deploy new virtual machines on the targeted host. To install the exacqVision Virtual
server, complete the following steps:
1.
2.
Open the vSphere Client and connect to your vCenter management server. If you do not have a vCenter server, connect
directly to your host.
From the File menu, choose Deploy OVF Template… . This opens the Deploy OVF Template wizard.
3.
4.
5.
6.
7.
8.
Click the Browse button and navigate to the downloaded exacqVision Virtual server. Click Next.
Click Next on OVF Template Details. Note the disk-size requirements.
Read and accept the license agreement. Click Next.
Rename the server if desired. Click Next.
Select the datastore you want to deploy to. Click Next.
Choose the type of provisioning. If you will modify the storage disk, choose thin provisioning to reduce the installation time.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 3 of 44
4/12/2013
9.
On the Network Mapping screen under Destination Networks, choose the network containing the exacqVision client.
10. Under Networking Properties, enter the Gateway, DNS, IP Address, and Netmask. If you prefer to use DHCP to populate
these values, click Next. This information can easily be changed from the web interface after deployment.
11. Under Ready to Complete, review the Deployment settings to make sure the information is correct. Select Power On After
Deployment if you want to start your server immediately (if you do not need to modify the default CPU, RAM, NIC, or disk
settings).
12. Click Finish.
You can watch the progress of the deployment in the Recent Tasks screen.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 4 of 44
4/12/2013
5 exacqVision Web Service
exacqVision Web Service is enabled on exacqVision Virtual Appliance by default. The Web Service requires increasing the number of
CPU cores in the appliance to six and increasing the memory to 6GB. It is recommended that you install a second exacqVision Virtual
Appliance specifically for Web Service to allow better VMware DRS load balancing.
Access the web interface by opening a browser with the server’s IP address and port 8080. If the server will be used exclusively for
the Web Service, open a command line interface (CLI) session and enter sudo rc to invoke the Exacq Linux Utility Script to unload the
exacqVision server.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 5 of 44
4/12/2013
6 Upgrading Hardware
The exacqVision Virtual Appliance is deployed with minimal hardware requirements. As your system grows, the virtual machine must
be monitored, and resources might need to be added.
NOTE: Adding network adapters (see section 10) and adding virtual disks (see section 9) do not require a restart of the server.
Adding CPUs and memory requires a shutdown of the system.
1.
2.
From the vSphere Client open the VMs and Template View, right-click on the exacqVision Virtual appliance.
From the pop-up menu, choose Power and then Shut Down Guest. This shuts down the server appropriately.
text
3.
After the server is shut down, right-click the server and choose Edit Settings. Alternatively, you can simply click on Edit
Virtual Machine Settings under Basic Tasks on the virtual machine’s Getting Started tab.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 6 of 44
4/12/2013
4.
To add memory, change the Memory value. To do this, select the Resources tab and change the reservation value to the
new memory value. Note that without reserving the memory, other virtual machines can utilize memory allocated to this
virtual machine if the host of the resource pool the virtual machine resides in becomes constrained.
5.
To add CPU cores, select the Hardware tab, click CPUs, and change the number of cores per socket. The total number of
cores should display the number of virtual sockets times the number of virtual cores.
NOTE: You can reduce the reservation of either memory or CPU resources without shutting down the server. If you have
resources to spare, it is better to allocate more memory or CPU resources and then set reservations levels to meet the
demand of the server during periods of the heaviest recording and viewing.
6.
Click OK to commit the changes, and then review the reconfiguration in the Recent Tasks pane at the bottom of the client.
7.
Click on Power on the virtual machine under Basic Tasks.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 7 of 44
4/12/2013
7 Configuration
The basic configuration of the exacqVision Virtual Appliance can be managed from the web interface and the console screen. To
access the console screen, open the VMs and Template view and select the newly deployed exacqVision Virtual Server. Right-click on
VM and choose Open Console. A blue screen appears with the access information similar to the following:
To configure the exacqVision Virtual Appliance, complete the following steps using the previous illustration as a guide:
1.
2.
3.
4.
5.
Enter the address shown on your console screen into a web browser to access the web interface.
This portion of the console screen shows the administrator login account and password.
This portion of the console screen shows instructions for returning to this screen from the CLI.
This portion of the console screen allows you to log in to the CLI.
This portion of the console screen is the CLI interface for setting the time zone. (It is preferred that you use the Web
interface for these settings.)
Open a Web browser and enter the address found using step #1 above. Be sure to include the portion of the address with a colon
and port 5480. If your browser gives you a security warning regarding the website security certificate, click Continue to this Website.
Log in with the username “admin” and the password “admin256.”
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 8 of 44
4/12/2013
The Web interface has two management tabs, as shown in the following illustration:

The System tab allows you to restart or shut down your server in a safe manner. Click the Time Zone button to set the
current time zone (the default time zone is UTC).
NOTE: The client and server can be in different time zones, but the server must be in the correct local zone.

The Network tab displays the current network configurations. You can click on the address button to change the address
from DHCP to static. From this screen you can also set the hostname and specify and Alternate DNS Server.
NOTE: If you change the address, you will lose connectivity to the web interface when you save the settings. You can then
type the new address in the browser to reconnect to the web interface. Do not change the IP address and hostname at
the same time, as the hostname change will not apply. After you change the IP address, you can then log back into the
web interface with the new address and then change the hostname.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 9 of 44
4/12/2013
8 Troubleshooting and Monitoring
The exacqVision Virtual Server provides a management GUI interface in the console screen for assisting troubleshooting tasks. The
server has a useful Linux script that can be accessed by entering rc from the CLI.
To access the GUI interface, open the console screen and choose Login. At the prompt, log in as “admin” with the password of
“admin256”. At the command prompt, type startx to open the Desktop interface, as shown in the following illustration:
1
4
2
3
5
The Desktop interface contains the following options:
1.
2.
3.
4.
5.
This opens the Firefox web browser. This is useful for accessing a camera’s web management interface directly.
The exacq script icon contains Linux utility commands that help fix common Linux issues.
The power button allows you to log out of the GUI and return to you the CLI. In the CLI, type exit to return to the console
screen.
Disk Utility (on the System menu under Administration) provides a graphic interface of mounted disk.
System Monitor (also on the System menu under Administration) provides statistics on CPU, memory, and network.
To change the console screen resolution from default 800 x600, open the Systems menu and select Preferences and then Monitors.
You can also change to a CLI by clicking Terminal under Applications – Accessories.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 10 of 44
4/12/2013
Virtual Machine
Because the exacqVision Virtual server is a virtual machine and shares server host hardware, performance issues can be identified
using the VMware client. To do this, start your client and open up the Host and Clusters view. From the Inventory pane, click on the
server host the exacqVision Virtual Server is running on and click the Virtual machines tab. This screen can quickly provide the
following information about the state of your servers:
1
2
3
4
5
6
7
8
1.
Name. For better organization, match your server name with the name of server in your exacqVision Client.
2.
State. Verify that server is powered on. If it is not, right-click the VM and select Power On.
3.
Provisioned Space. View the amount of space is allocated to the VM. If recording history is too short, additional virtual disk
might need to be added (see Section 10).
4.
Used Space. View how much of the allocated space is in use.
5.
Host CPU – MHz. View how much of the CPU resources your virtual machine is consuming from the host. The default CPU
reservation is 2000 MHz; if this value is close to 2000, increase your virtual machine’s CPU count or reservation (see
Upgrading Hardware).
6.
Host Mem – MB. This is the actual value of memory being allocated to the server. This number indicates the memory
reservation and host memory overhead.
7.
Guest Mem – %. The bar graph shows the percentage of memory in use. If this is consistently 80% or higher, consider
allocating additional memory to the virtual machine (see Upgrading Hardware).
8.
Notes. Add descriptive information about each virtual machine by editing Annotations, found under the Summary tab of
the virtual machine.
Additional virtual machine performance monitoring information can be found in section 11.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 11 of 44
4/12/2013
9 Configuring Virtual Disk Storage
The default deployment of the exacqVision Virtual Appliance (EVA) includes only a 100GB hard drive. In the video storage world, this
is not very much storage; as you increase the number of camera, storage must also be increased. You can use the Exacq
Configuration Calculator to determine the amount of disk size that you will need. For example, a typical Axis 1.3 MP camera
recording 15 frames per second 30% of the time would require about 10 days of video storage on 100GB of storage space.
Do not remove the 100GB default drive from the EVA until after you start the server for the first time. After the server is online, you
can proceed with removing or adding new drives using the vCenter client and the appliance command line interface. If you remove
the default storage drive before starting the server, the server will lock up. If you don’t plan to use the 100GB drive, you can just thin
provision it. It is not necessary to remove this drive before adding new drives targets because exacqVision Client allows you to
determine which attached drives you want to save data to.
The exacqVision data disks are mounted by label, which is the same as the mount point for each drive. If a new drive is manually
added, modify the etc/fstab file using the same labeling format as the default disk drive:
LABEL=/mnt/export /mnt/export ext4 defaults 0 0
Because the data drive will consume most of the space of any drive, there is no need to partition the data drives. Linux allows the
laying of a file system directly onto the disk. Only the system drive /dev/sda is partitioned.Virtual hard disk 2 is dedicated to
exacqVision export of video files and is 10GB by default. Exported files are not automatically deleted; thus, available space on the
drive must be monitored. The instructions in section 1, “Expanding Virtual Disk Storage” can be used to increase storage availability.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 12 of 44
4/12/2013
Expanding Virtual Disk Storage
Virtual hard disk 3 is configured for video storage and is formatted with the ext4 file system. This virtual disk can grow to a VMware
maximum of 2TB by editing the settings using the VMware client, and this does not require a reboot of the EVA. The EVA system will
scan the drive every five minutes and automatically register the new space after the scan. Adding a new drive to the EVA requires
the exacqVision server service to be restarted temporary, which suspends video recording and viewing.
VMware allows the simple expansion of a virtual disk. After the disk size is increased on the hypervisor server, the operating system
must be adjusted to see the new storage. These instructions are intended for expanding the default 100 GB store, but they can be
used to expand any virtual disk that is added to the EVA.
To expand the virtual disk, complete the following steps:
1.
Right-click the exacqVision Virtual Appliance and choose Edit Settings. Click on Hard Disk 3 and change the provisioned size
to 2048 GB in the right pane. Note the SCSI host device node number; you will need this when you rescan the SCSI host. In
this example, this important number is 0:2.
2.
After saving the new value, open the console screen of the EVA and press Enter to log in to the command line interface (CLI)
with the admin account. You can grow your disk incrementally to a maximum of 2TB.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 13 of 44
4/12/2013
3.
Click on the console screen and wait for the server to display the default console screen.
4.
Click in the console screen and press Enter to go to the CLI. Log in as admin.
5.
The devices on the SCSI bus are listed in the /sys/bus/scsi/devices/ directory. In the default deployment of the EVA, three
devices correspond to the three disk and are listed by a four-digit number in a busid:channel:target:LUN format. For our
100GB data disk, the device number would be 2:0:2:0. The middle two numbers match the virtual device node from step 1.
Rescan the this hard drive by using this command:
echo 1 | sudo tee /sys/bus/scsi/devices/2:0:2:0/rescan
6.
Format the expanded space using resize2fs. Existing data will not be erased; only the new space will be formatted. This
process requires an extended amount of time, depending on how much you expand the volume:
sudo resize2fs /dev/sdc
The output should look similar to this……
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 14 of 44
4/12/2013
7.
After the drive is finished resizing, run the sudo df /dev/sd* command. The drive should now show up with the added
space listed under 1K-blocks. In our example, the drive increased by 100GB, and we now see 206424760 1K-blocks.
8.
Type logout to exit the CLI
9.
Open exacqVision Client and click on the Config (Setup) icon
minutes, the data drive should now report the additional disk space.
www.exacq.com
. Navigate to storage in the site tree. Within five
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 15 of 44
4/12/2013
Adding Virtual Disk to exacqVision Virtual Appliance.
To dynamically add a virtual disk to the EVA, complete the following steps:
1.
Right-click the EVA and click Add. Choose Hard Disk and then click Next.
2.
On the Select a Disk screen, click Next to create a new virtual disk.
3.
On the Create a Disk screen, specify the new disk size (up to 2TB). Exacq recommends provisioning the drive as Thick
Provision Eager Zeroed.
NOTE: Depending on the size of the new disk, this process could require an extended period of time and might
negatively affect the performance of a host. Thin provisioning is not an ideal choice, as the drive will quickly become
full of recorded data, and performance might lag as the thin-provisioned drive expands. You can choose to keep the
new drive with the virtual machine, or you cay choose any available datastore presented to the host. Note that video
traffic creates a significant load on disk writes, so you will need to monitor your data store to ensure acceptable
performance.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 16 of 44
4/12/2013
4.
On the Advance Options screen, note the Virtual Device Node ID and click next.
5.
On the Ready to Complete screen, review the options and click Finish.
6.
Click OK to close the Edit Setting screen and create the disk. You can watch the progress of the disk creation in the
Recent Tasks Pane. Do not proceed until the Reconfigure Virtual Machine task is complete.
7.
Click on the console screen and wait for the server to display the default console screen:
8.
Click in the console screen and press Enter to display the CLI. Log in as admin.
9.
Rescan the SCSI host on host2 to discover the new drive using the following command:
echo “- - -“ | sudo tee /sys/class/scsi_host/host2/scan
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 17 of 44
4/12/2013
A new device should now be displayed in the /sys/bus/scsi/devices directory as 2:X:X:0, where X:X matches the virtual
device node of the drive you just created.
Type the command ls –l /dev/sd* to produce an output with the last line showing the new drive called /dev/sdd:
10. The exacqVision video storage should be mounted in the directory /mnt/edvr/# where each # is associated with a
label attached to a storage device. Use the ls/mnt/edvr command to view the number of mount points. You must
create a new mount point, incrementing # to the next number. The full path of this new directory will be used as the
label for the device.
ls /mnt/edvr
sudo mkdir /mnt/edvr/2
ls /mnt/edvr
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 18 of 44
4/12/2013
11. Format the new drive using the mkfs command in ext4 format:
sudo mkfs.ext4 –F –L /mnt/edvr/2 /dev/sdd
NOTE: The –L portion creates a label for the new drive and should be designated as the path to the new mount point
created earlier.
The drive then formats the inode tables. Depending on the size of the drive, this could require a few minutes.
12. The fstab file must be modified to include the new disk so that it will be mounted automatically after a restart. Open
this file by using the Nano text editor, as follows:
sudo nano /etc/fstab
13. Under the last line, insert a label defining the new label used in the mkfs command. The mount point should be the
full path to mount point created previously. Be sure to follow the same syntax, as Linux is case-sensitive.
14. Press CTRL+O and then press Enter to save the file. Press CTRL+X to exit the editor.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 19 of 44
4/12/2013
15. Re-read the fstab file by typing this command:
sudo udevadm trigger
16. Mount the drive using the drives label, which is the same as the mount point:
sudo mount /mnt/edvr/2
17. Verify the new drive by using the command df. The last entry should be your new drive. In this example, we added a
500GB drive mounted at /mnt/edvr/2.
18. The exacqVision Service must be restarted to see the new drive:
sudo /etc/init.d/edvrserver restart
19. Open up the exacqVision Client, click on the Config (Setup) icon
should be displayed and enabled for storage.
www.exacq.com
, and navigate to Storage. The new drive
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 20 of 44
4/12/2013
Removing a Virtual Disk
If you want to remove a disk, the drive must first be removed from the operating system and then removed from the virtual
machine. Failure to remove the disk from the operating system first will result in the operating system freezing when started.
The 100GB storage drive is also provisioned to be on the same datastore as the server. This might not be desirable for your
specific storage setup. If this is an initial setup, and data has not been written to the default disk, Exacq recommends thin
provisioning the disk and simply disabling the disk as a recording source in exacqVision Client.
In the following example, a 500GB storage drive is added on an external ISCSI storage server and the 100GB default drive is
removed from the same datastore location as the EVA.
NOTE: After storage is removed, any video previously stored on the drive is lost. Any data that you want to preserve or search
must need to be migrated to another drive before deleting the drive.
1.
2.
3.
4.
Open exacqVision Client, click on the Config (Setup) icon
, and navigate to Storage. Deselect the 100GB drive to
stop recording to it. Click Apply.
Click in the console screen and press Enter to display to the CLI. Log in as admin.
If data must be migrated or deleted, you must do that now.
Unmount the drive by its label (which is the same as its mount point) using this command:
sudo umount /mnt/edvr/1
Do not remove the mount point directory. If you are not sure which drive is the correct drive, use the df command to
view the mounted drives. In this case, the drive is sdc mounted on /mnt/edvr/1.
5.
The fstab file must be modified to remove the entry for the disk. Open this file by using the Nano text editor as follows:
sudo nano /etc/fstab
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 21 of 44
4/12/2013
6.
Locate the drive listed by its label. Use the arrow keys to move to the front of the line and press Delete until the entire
line is deleted and the bottom line is moved up.
7.
Press CTRL+O and then Enter to save the file. Press CTRL+X to exit the editor.
8.
Delete the device from the operating system using the following command:
echo 1 | sudo tee /sys/block/sdc/device/delete
9.
The drive can now be removed from the virtual machine. Right-click on the EVA appliance and click Edit Settings. Click
on the target hard drive to be removed and then click on the Remove button. Click OK.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 22 of 44
4/12/2013
You have now successfully removed the drive from your appliance. To reclaim the space, use vSphere Client to browse the datastore
where your appliance was deployed to and delete the file ending in “_2.vmdk” that has a provisioned size of 104,857,600.00 KB.
NOTE: exacqVision Client will display the removed drive with a status of Drive not Found.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 23 of 44
4/12/2013
Working with Logical Volumes
VMware has a hard limit of 2TB minus 512 bytes. Disk space can be added to the server by mounting individual disk drives up to the
2TB limit. As you expand your system, or decide to deploy an EVA that requires a large amount of disk space, it might be easier to
create a LVM combining multiple 2TB virtual disks. Your physical storage should be protected by a RAID level, so there is no need to
create a RAID group with the defined LVM, as this would decrease performance . When you deploy a LVM, it is important that all of
the LVM disks are from the same VMware-defined datastore.
1.
Right-click the EVA and click Add … . Choose Hard Disk and then click Next.
2.
On the Select a Disk screen, click Next to create a new virtual disk.
3.
On the Create a Disk screen, specify the new disk size (up to 2TB). Exacq recommends provisioning the drive as Thick
Provision Eager Zeroed.
NOTE: Depending on the size of the new disk, this process could require an extended period of time and might
negatively affect the performance of a host. Thin provisioning is not a good choice, as the drive will quickly become
full of recorded data. You can choose to keep the new drive with the virtual machine, or you cay choose any available
datastore presented to the host. Note that video traffic creates a significant load on disk writes, so you will need to
monitor your data store to ensure acceptable performance. Also, make sure your network is appropriate for managing
the increase data streams.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 24 of 44
4/12/2013
4.
On the Advance Options screen, note the Virtual Device Node ID and click next.
5.
On the Ready to Complete screen, review the Options and then click Finish.
6.
Repeat these steps for each disk you want to add.
7.
Click OK to close the Edit Setting screen and create the disk. You can watch the progress of the disk creation in the Recent
Tasks Pane. Do not proceed until the Reconfigure Virtual Machine task is complete.
8.
Click on the console screen and wait for the server to display the default console screen:
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 25 of 44
4/12/2013
9.
Click in the console screen and press Enter to display the CLI. Log in as admin.
10. Rescan the SCSI host on host2 to discover the new drives:
echo “- - -“ | sudo tee /sys/class/scsi_host/host2/scan
The new drives should now show up in your /sys/bus/scsi/devices directory as 2:X:X:0, where X:X matches the Virtual
Device Node of the drives you just created.
Typing the command ls –l /dev/sd* now produces an output with the last line showing the new drives as sdd and sde.
11. Using the pvcreate command, create a physical volume label out of the virtual disk. List each disk that will be used in the
volume, separated by a space.
sudo pvcreate /dev/sdd /dev/sde
12. Create a volume group and give it a name that references the datastore the physical drives are on, with an incremented
number. List every physical volume that you just created, separated by a space. If you already have volume groups on the
server, use the vgcreate command to determine the next increment number:
sudo vgcreate ExacqStorageServer01 /dev/sdd /dev/sde
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 26 of 44
4/12/2013
13. Create a logical volume using all of the space. The command is formatted as follows:
sudo lvcreate –L (size of Volume) (name of volume group) –n (Logical Volume Name)
In most cases, you will be allocating all the free space in the logical volume to video storage. You can accomplish that by
substituting “-L (size of Volume) in the command above with “-l |100%FREE.” In this example, the command would look like
this.
sudo lvcreate –l +100%FREE ExacqStorageServer01 –n Video_Storage_01
If you do not want to use all the space, you must specify the size. In this example, two 2TB volumes have been added at
3999.99G. To determine smaller volumes, use the figure 99.997 for every 100GB allocated in a disk. If you are close to using
all of the space, the system will round up to use all the space.
The volume group name should be the name that you provided earlier. You could use a name like Video_Storage_XX,
where XX is an incremental number for the syntax of the LV name:
sudo lvcreate –L 3999.99G ExacqStorageServer01 –n Video_Storage_01
14. You can now view you new volume using the lvs command:
15. Format the Logical volume using the ext4 format. There are a number of information-gathering steps that should be done
first before proceeding with the format.
The new device created will exist in a new directory with the name of the volume group. Use the lvscan command to get
the full path.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 27 of 44
4/12/2013
16. The exacqVision video storage should be mounted in the directory /mnt/edvr/#, where each # is associated with a label
attached to a storage device. Use the ls/mnt/edvr command to view the number of mount points. You must create a new
mount point incrementing to the next number. The full path of this new directory will be used as the label for the device:
ls /mnt/edvr
sudo mkdir /mnt/edvr/2
ls /mnt/edvr
In this example, the new label would be /mnt/edvr/2. Knowing where the device is located and the label name to use,
format the new volume. Note that you are creating very large volumes, so this will take an extended time to write out the
inode tables. There are 16000 inode tables for each 2 TB drive.
sudo mkfs.ext4 –L /mnt/edvr/2 /dev/ExacqStorageServer01/Video_Storage_01
The output should look similar to this:
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 28 of 44
4/12/2013
17. Edit the fStab to include the new label. Be sure to note case sensitivity. Open the fstab file using nano:
sudo nano /etc/fstab
Under the last line, insert a Label defining the new label used in the mkfs command. The mount point should be the full
path to mount point created previously.
18. Press CTRL+O and then Enter to save the file. Press CTRL+X to exit the editor.
19. Re-read the fstab file by typing this command:
sudo udevadm trigger
20. Mount the new volume by specifying the full device path name and mount point:
sudo mount /mnt/edvr/2
21. Use the df command to view the new disk. Notice that you now have a 4 TB volume
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 29 of 44
4/12/2013
22. The exacqVision Server must be restarted to see the new drive:
sudo /etc/init.d/edvrserver restart
Open exacqVision client and click the Config (Setup) icon
. Under your server in the site tree, open Storage and view
the new 4TB volume. It should be listed as Enabled and ready to receive video storage.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 30 of 44
4/12/2013
10 Configuring Additional Networking
A common configuration isolates video traffic from management traffic for either security reasons or to isolate video traffic on a
physically separate network. The exacqVision Virtual appliance is deployed with only one interface (eth0). Before adding interfaces,
start the appliance for the first time. The first network connection’s MAC address is used for licensing the server in the client. This
interface should always be configured for your management network.
If you want physically separated networks attached to your VMware host, configure a new standard switch and connect it to one or
more dedicated virtual machine NICs. A VMkernal Port is not required for the switch.
If a physical separate network is not available and your network supports VLANs, create a Camera Virtual Machine port group using
the same physical interfaces as the management network, and then assign to a security VLAN ID on your network.
Configure a second port group for the management interface and use VLAN ID 4095, which accepts all VLAN traffic. Connect the
deployment interface to this port group. After the virtual machine has been deployed, add a second Ethernet interface and attach it
to the Camera Port group.
Either scenario can support multiple physical adapters in order to provide redundancy for the virtual switch.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 31 of 44
4/12/2013
After the appliance is running and the IP address of eth0 has been applied, a second Ethernet adapter can be configured using the
following steps:
1.
2.
3.
4.
From the vSphere Client, right-click the exacqVision Virtual Appliance and choose Edit Settings.
Click the Add… button.
Choose Ethernet Adapter and click Next.
Change the adapter type to E1000. Click on the Network Connection drop-down list and choose the network that will be
used for camera traffic. Make sure Connect at Power On remains selected, and then click Next.
5.
6.
7.
Click Finish to close the wizard and click OK to complete the editing of the virtual computer.
Open a console screen and log in using the admin account.
Configure the new interface using the vami_set_network command. This allows you to set a static IP address on the new
interface that was just added.
If your exacqVision Virtual Appliance is Version 5.4 or later, then you can run the Vami configuration script:
sudo /exacqscripts/VamiNet.sh
Otherwise, use this command:
sudo /opt/vmware/share/vami/vami_set_network <eth#> STATICV4 <ip address> <net mask> <gateway>
In the following example, we add one new adapter, which is eth1. The camera network is in the 10.10.x.x network with a
Class C subnet mask. The new adapter address is 10.10.1.30, and the gateway is 10.10.1.254. The full command would be:
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 32 of 44
4/12/2013
sudo /opt/vmware/share/vami/vami_set_network eth1 STATICV4 10.10.1.30 255.255.255.0 10.10.1.254
The output would look similar to the following:
8.
If a DHCP server is located on the management network, eth0 lose its fixed IP address. Reset the interface using these two
commands:
sudo ifdown eth0; sudo ifup eth0
9.
Open a web browser and navigate to the appliance’s web interface. Log in as admin and click on the Network tab. Under
Status, note that the two adapters are correctly configured to two separate networks.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 33 of 44
4/12/2013
11 Performance Monitoring
Virtual Machines share host resources, so performance issues can be cause by either the virtual machine’s not having enough
resources or by reaching the physical resource limit on the the virtual machine’s host The vSphere client provides some several tools
to evaluating performance problems on both the virtual machine and host.
VM Performance Tab
There are several indicators that should be checked if the exacqVision Virtual appliance has performance issues. You must consider
the CPU performance of both the virtual machine and the host. The Performance tab can provides a quick indicator of performance
issues.
To monitor virtual machine performance, open the VMware client and navigate to Host and Clusters. Select the exacqVision Virtual
machine in the Inventory pane, select the Performance tab, and choose the 1 Day time range. From the Overview screen, view the
CPU (%) graph. If the average is above 80%, consider adding a virtual CPU to the machine by editing the machine settings.
Also note the memory balloon and swap-out rates. These indicators should be at or near zero. Values in these charts indicate
memory limit placed on the virtual machine or memory overcommitment. A value of 6% or less of memory free on the host indicates
overcommitment of memory resources.
Scroll down the page to view the memory percentage. The exacqVision Virtual server defaults to 2GB of RAM with a reservation of
2GB. If this value averages above 80%, consider adding memory resources to the virtual machine. If this is a small deployment and
this number is consistently below 50%, you might be able to reclaim some host resources by lowering the memory reservation.
NOTE: Usage of the exacqVision Web Service will dramatically increase CPU and memory demands on the virtual machine.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 34 of 44
4/12/2013
On a healthy system, Disk KBps and Network Mbps should roughly look the same. The exacqVision Virtual machine is normally one
of the largest resource users of host network and datastore traffic. If network latency becomes an issue, consider adding or
dedicating physical NICs for VMS network traffic.
Host Performance
The virtual machine host must have enough memory and CPU resource for each virtual machine that is assigned to it. You can
overprovision CPU and memory resources, but when the hosted virtual machine exceeds the physical resource capacity of the host,
all of the virtual machines will experience performance issues as VMware pulls resources from running virtual machines.
The VMware host can quickly be assessed by opening the Hosts and Clusters view and clicking on the host in the Inventory pane.
Click on the Summary tab and observe the statistics under Resources. Host resources are displayed with a bar meter and provide a
quick view of current and CPU and Memory loads. Note that CPU usage should be less than 80%. Multiply the number of cores by
the core GHz to get your total CPU capacity. Multiply by .8 and verify that the CPU usage number is lower:
8 x 2.393 GHz = 19.144 (total GHz) *.8 = 15.3152 (80% capacity ) > 4216 MHz ( current Load )
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 35 of 44
4/12/2013
Observe the Memory Usage counter and verify that is it stable. If you are near your memory capacity during peak operation,
consider moving virtual machines to a different host and adding memory to the host. More detailed information can be viewed by
opening the Performance Tab. Some of the key indicators for memory include the Balloon and Swap charts. Both of these values
should be 0, and values here indicate that the host is borrowing and using disk for memory, which negatively affects performance.
Hosts share physical network connections among virtual machines. Depending on the number of cameras recording to the VMS,
significant amounts of bandwidth might be needed. Scroll to the bottom and observe the Network (MBPS) usage. This chart provides
a quick overview of the average rate at which data is transmitted and received across all NIC instances connected to the host. High
values here might indicate that packets are being dropped, which could indicate a bottleneck in the network.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 36 of 44
4/12/2013
The VMS system writes large amounts of data to datastores attached to the host. Bad disk performance often shows up as high CPU
usage in your virtual machine, as the disks are not able to keep up. Disk latency is a good indicator of disk performance issues. Click
on the Advanced button and then Chart to open the Customize Performance Chart window. Under Chart Options, choose Disk. Then
choose the specific datastore object your exacqVision Virtual server is writing to. Under Counters, check Kernel Write Latency and
Physical Device Command Latency. Click Apply to view the statistics.
The Physical Device Command Latency should be less than 20ms, and the Kernel Command Latency should be less than 3ms. Higher
values indicate the datastore does not have the IOPS capacity to keep up with the load. Consider adding datastore capacity or
migrating virtual machines to a different datastore to lessen the load.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 37 of 44
4/12/2013
12 Upgrading exacqVision to a New Release
The exacqVision Virtual appliance can be upgraded the same as any physical appliance. However, as improvements are made to the
appliance, you might want to upgrade the appliance itself. You can download the latest version of exacqVision Virtual at
https://www.exacq.com/products/exacqVision_virtual.php.
To preserve current video data, you must have rights to the VMware datastore where the exacqVision Virtual drives are mounted. If
you have already downloaded the latest version of exacqVision Virtual to the location where your VMware is located, this entire
process can be completed in less than 30 minutes.
Prerequisite steps
1.
2.
3.
4.
5.
Print the exacqVision Virtual Pre-Migration Data Sheet to record your system settings. NOTE: Depending on your
configuration, you might not have configuration data.
Using the VMware client, the version number of your appliance can be found in the summary tab at the top of the General
box. If you have exacqVision Virtual version 5.2.1.36311, download the pre-migration script file found on the exacqVision
web site.
Download exacqevaback.sh and copy it to the /exacqscripts directory under the root directory. If you have Internet access
to your virtual environment, you can browse to the link using Firefox from your appliances GUI interface.
Make the file executable by opening a Terminal interface and running sudo chmod 0775 /exacqscripts/exacqevaback.sh.
This Script copies your configuration files to a folder on mount point /mnt/edvr/1.
Execute the command by running sudo /exacqscripts/exacqevaback.sh.
Removing Old Server
1.
2.
3.
4.
5.
6.
7.
8.
Open the vSphere Client and navigate to your exacqVision Virtual server.
Right-click the server and choose Open Console.
Log in into the console as an admin
Run sudo exacq to open the utility menu.
Choose option 3 to stop the exacqVision Server.
Close the console screen.
Right-click on your server and choose Power | Shut down Guest.
When the green arrow turns off on the VM, right-click and choose Remove From Inventory.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 38 of 44
4/12/2013
9.
Press Ctrl+Shift+D to enter the Datastore view. In the navigation pane, choose the datastore recorded in line 10 of the PreMigration Data Sheet. Click Browse Datastore.
10. Select the root of the datastore and click the new folder Icon. Create a new folder with name of your server followed by the
word “disk. This will be the location used to save your virtual data disk.
11. Navigate to your virtual machine and click on the Type header to sort by Provisioned Size. You will be moving the data
virtual disk the folder you just created. Do not move the two 10GB disks.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 39 of 44
4/12/2013
12. Right-click on the first drive and choose Move to … .
13. Click Yes to confirm the move and navigate the folder you created for your disk on the same datastore. The move should be
instantaneous if on the same datastore. Use the CTRL key to select multiple disks and move all at once.
14. When all the disks have been moved, right-click on the virtual server folder and click the red X to delete.
15. Minimize the datastore browsers window.
Deploy The New Server
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
In the vCenter client, press Ctrl+Shift+H to enter the Hosts and Clusters view.
Select File and choose Deploy OVF Template.
Browse to your new downloaded exacqVision OVA file. Click Open and then Next.
On the OVF Template Details screen, click Next.
Click Accept on the End Users License Agreement, and then click Next.
Enter the exact name of your old server.
Browse to the same data center from line 7 of the Pre-Migration Data Sheet. Click Next.
Click on the cluster from line 7 of the Pre-Migration Data Sheet. Click Next.
Choose the Host from line 8 of the Pre-Migration Data Sheet. Click Next.
Choose the datastore from line 10 of the Pre-Migration Data Sheet. Click Next.
Choose Thin Provision and click Next.
Select the drop-down under Destination Networks and select the Network Adapter 1 Connection Network label recorded
on line 9 of the Pre-Migration Data Sheet. Click Next.
13. Enter the Networking Properties for eth0 from line 4 of the Pre-Migration Data Sheet. Click Next.
14. Do not check Power On After Deployment. Click Finish.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 40 of 44
4/12/2013
Reconfigure The New Appliance
1.
2.
3.
Right-click the newly deployed server and choose Edit Setting.
Select Hard Disk 3 and click Remove.
When you see a list of Removal Options, select Remove From Virtual Machine and Delete Files From Disk. Click OK to
apply the changes.
4.
Open the datastore browser window and click the refresh icon at the top. The newly deployed appliance is displayed
with the same name as the old server.
5. Open the disk folder created earlier and select all the migrated disks. Right-click and follow the instructions.
6. After all the disks are moved, click on the disk folder and click the red X to remove the folder. Close the Datastore
Browser.
7. Right-click your appliance and select Edit Settings.
8. Click on Memory and change it to the value from line 9 of the Pre-Migration Data Sheet. Click Next.
9. Click on CPUs and change it to the value from line 9 of the Pre-Migration Data Sheet. Click Next.
10. Click on Network Adapter. On the right side, under MAC Address, click Manual. Enter the last three octets from the
address recorded on line 3 of the Pre-Migration Data Sheet. Click Next..
NOTE: Be sure to put a colon (:) between each octet. Do not erase the first three octets.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 41 of 44
4/12/2013
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
Click the Add button, choose Hard Disk, and click Next.
Select Use an Existing Virtual Disk and click Next.
Click Browse and navigate to the datastore recorded on line 10 of the Pre-Migration Data Sheet. Click Open.
Navigate to the newly deployed server and click Open.
Expand the Name column and choose the disk ending with “2.vmdk.” Click OK, Next, Next, and Finish.
Repeat the process starting with step 11 for each disk that was migrated. Be sure to increment the disk #.vmdk each
time (3.vmdk, 4.vmdk, and so on).
Click the Resource tab and change values for CPU and Memory reservations to the values from line 11 of the PreMigration Data Sheet.
Click OK to close the Virtual Server Properties. After it is reconfigured, right-click on the virtual server, select Power,
Power On. Your server will copy all saved information and remount all data drives.
If you have a second network adapter, right click your Virtual Server and choose edit. Click Add. Choose Ethernet
Adapter and click Next. Change the Adapter Type to E1000. In the drop-down list under Network Connection, select the
network listed for Eth1 from line 9 of the Pre-Migration Data Sheet. Click Next and Finish. Click OK to save the settings.
Right-click on the Appliance and choose Open Console. Log in to the console using the admin account.
In Terminal, run sudo /exacqscripts/VamiNet.sh to set the network adapter address. Exit Terminal.
In exacqVision Client, select the Config (Setup) icon and browse to the server, which should have reconnected
automatically. The version number in the right pane should be incremented to the latest EDVR major release version.
Select System. Verify that your license and system name are correct.
Select the Date/Time tab and verify that the Time Zone is correct.
Select the Network tab and verify that the network setting for each network interface is correct.
Select the ActiveDirectory/LDAP tab and verify the settings are correct (if applicable).
Select the Storage icon and verify that all the disks are displayed and listed as Healthy.
Select Add IP Cameras and verify that all the cameras have reconnected.
Remove the evabackup directory.
Right-click on the Appliance and Choose Open Console.
Log in to the console using the admin account.
In Terminal, run sudo rm –rf /mnt/edvr/1/evabackup.
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 42 of 44
4/12/2013
13 Pre-Migration Data Sheet
Use this sheet to record important information regarding your server before migration. For the Virtual Appliance hardware setting,
note any changes that from the default (shown in parentheses).
1.
Open exacqVision client and connect to the server. In the navigation pane, find your server. In the system information
screen, note the version number.
Current Version Number ___________________
2.
3.
Select System in the navigation Pane. Click the Export button and save your configuration.
Under License, click the Export button and save your license file. Record the last three octets of the license MAC address
under the license number.
MAC Address 00
4.
- 50 - 56 - ________ - ________ - ________
Select the Network tab and record the network settings.
ETH 0
ETH 1
IP Address
Netmask
Gateway
Primary DNS
5.
Select the Date/Time tab and record: Time Zone _______________ Time Server __________________
6.
Select the ActiveDirectory/LDAP tab and record (if applicable):
LDAP Server Address ___________________
LDAP Schema
_____________________________
LDAP Base DN __ ______________________________________________________________________
LDAP Binding DN _________________________________________________________________________
7.
Open your vSphere Client and navigate to Hosts and Clusters (CTRL+SHIFT+H). Locate your server and record:
VM Name
Data Center
8.
_________________________________
___________________________ Cluster
____________________________
Click on the Servers Summary tab and record the host:
Host IP Address __________- __________-__________-__________
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 43 of 44
4/12/2013
9.
In the navigation pane, locate your server, right-click, and choose Edit Settings. Under Hardware record:
Memory (2048) ______________________
CPU:
Number Virtual Sockets (2) ______ Number of cores per Socket (1) ________
Network Adapter1 Network Connection Network Label: _________________________________
Network Adapter2 Network Connection Network Label: _________________________________
10. Click Hard Disk 1 and note the Data Store under Disk File [in brakcets]
Data Store
____________________________________________________________
11. Click the resource tab and record:
CPU
Shares: (Normal) __________
Reservation: (2000) ___________
Limit: (Unlimited) ____________
Reservation: (2048) ___________
Limit: (Unlimited) ____________
Memory
Shares: (Normal) __________
www.exacq.com
+1.317.845.5710
+5255.56080817
+44.1438.310163
USA (Corporate Headquarters)
Mexico
Europe/Middle East/Asia
Page 44 of 44
4/12/2013