That IS the question. The V in this case stands for virtualization, specifically Microsoft’s Hyper-V. And after more time than I’d like spent trying to work through an issue I was encountering, my answer became no. At least for now. If you are up for an epic and slightly tragic story then read on.
Here is what happened. In getting my FPGA development environment setup, I followed the very detailed and straight forward installation instructions for Vivado Design Tools Suite. I also copied over the Digilent Board Files. Then, I followed the step-by-step getting started instructions to create my first project in Vivado. This involved several steps:
- Selecting the board type. As discussed here, I’ve chosen the Digilent Basys 3
- Copying in constraints files (we will talk about those in a later post)
- Creating a very simple Verilog file
- Synthesizing the bitstream (equivalent to compiling and linking)
- Connect to the device using the Hardware Manager in Vivado
- Deploy the project to the Basys 3
All went well until step 5. At that point, the Hardware Manager was unable to find my device. Here is the error message from the log window.
connect_hw_server -allow_non_jtag
INFO: [Labtools 27-2285] Connecting to hw_server url TCP:localhost:3121
INFO: [Labtools 27-2222] Launching hw_server...
INFO: [Labtools 27-2221] Launch Output:
****** Xilinx hw_server v2020.1
**** Build date : May 27 2020 at 20:24:38
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
ERROR: [Labtools 27-2220] Launch Error:
Unable to launch local hw_server executable.
ERROR: [Common 17-39] 'connect_hw_server' failed due to earlier errors.Hmm, so what’s going on here?
The first thought is that the USB cable drivers were not installed correctly. Similar to a great many embedded development boards, the Basys 3 uses an FTDI chipset for USB communications. If the drivers were not installed correctly, then you would expect that the board would not enumerate properly in the Windows Device Manager. So let’s check for that. The FTDI drivers should enumerate associated devices as USB Serial Converters.
Looking at USB controllers node in the Windows Device Manager we can see two USB Serial Converters listed, A and B. They are right at the bottom of the list of USB devices. So that is not it. But maybe the drivers did not install correctly. Let’s reinstall them just in case. Xilinx conveniently providers a command file script for reinstalling the cable drivers just in case you need it. For 64-bit Windows you would do the following from a Windows command prompt.
C:\>cd <Vivado install path>\data\xicom\cable_drivers\nt64
C:\>install_drivers_wrapper.batAfter a successful re-install of the required cable drivers, I again tried the connect to the board again from the Vivado Hardware Manager, no luck. Ok, let’s take a closer look at the driver setup for the FTDI device. In the image below we can see that the driver provider is FTDI, check. And further we can see that the driver was signed by Microsoft Windows Compatibility. That all seems reasonable.
But let’s click on the Driver Details just to see that there at least one or more driver files associated with the device. Doing so shows that there are in fact a number of driver files that are mapped to this device. Looking more closely at the file names shows that they are named with variations of FTxxx which probably makes sense for a set of FTDI drivers.
And finally, looking at the Device Details and the Bus Reported Device Description property we see that the device reports itself to Windows as a Digilent USB Device. That sure looks like Windows and the Basys 3 are communicating properly. At least as far as device recognition and enumeration. That would seem to rule out a fundamental issue with the board itself and the drivers.
One thing I did notice was that the date of the driver was 2017. One would hope that the Vivado 2020 installation would include the latest device drivers. But you never know, and so always good to check for updated drivers just in case. I went ahead and clicked on Update Driver and let Windows search for the latest drivers. In this case, these are in fact the latest device drivers that Windows can find for these devices.
It must not be the drivers
Taking a closer look at the error log from Vivado I note the message: Connecting to hw_server url TCP:localhost:3121. This is followed soon after by the message: ‘connect_hw_server’ failed due to earlier errors. So it seems that the Vivado Hardware Manager expects to be able to connect to a process called hw_server running on the localhost. And further, it believes that hw_server is listening for incoming connection requests on TCP port 3121.
A quick check of the Windows Task Manager shows that in fact no process called hw_server is running. A search of the Xilinx\Vivado directory tree for hw_server.exe shows that this file lives in <Vivado Dir>\bin\unwrapped\win64. Let’s run the server manually and see what that shows us.
D:\Xilinx\Vivado\2020.1\bin\unwrapped\win64\hw_server.exe
****** Xilinx hw_server v2020.1
**** Build date : May 27 2020 at 20:24:38
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
INFO: hw_server application started
INFO: Use Ctrl-C to exit hw_server application
Cannot create listening port: Socket bind error. An attempt was made to access a socket in a way forbidden by its access permissions
Who is using the ports?
Now this is interesting. Very interesting as I’ve spent a fair bit of time writing networking code. The hw_server starts up and then it reports that it cannot create a listening port because of a Socket bind error. Presumably the hw_server is trying to bind a socket to TCP port 3121 so that it can receive connection requests on that port. Now if you are not a network programmer this may not make a great deal of sense to you. But the thing to know is that of the 65,536 TCP ports available, only ports 0 through 1023 are reserved for special use by the Internet Assigned Numbers Authority (IANA). These are documented in Request for Comments (RFC) 1060.
So the hw_server should be able to bind to that port. That is, unless some other process has already bound that particular port. You can check for this by running the following from a Command Prompt as an Administrator: netstat -abp TCP | find “3121”. When I did this, I found no processes listening or connected to that port. So why did the attempt to connect fail? Could a Firewall rule be blocking the port? A quick check showed that this was not the case.
Who is blocking the ports?
Not certain exactly what the hw_server process is doing, I decide to write my own code to test a simple binding of a socket to port 3121. The key parts of that code is shown below.
int createListenSocket(int port)
{
errCode = E_NONE;
auto s = socket(PF_INET, SOCK_STREAM, 0);
SOCKADDR_IN sin;
sin.sin_family = PF_INET;
sin.sin_addr.s_addr = 0;
sin.sin_port = htons(port);
auto result = bind(s, (LPSOCKADDR)&sin, sizeof(sin));
if (result)
{
errCode = WSAGetLastError();
closesocket(s);
return errCode;
}
result = listen(s, SOMAXCONN);
if (result)
{
errCode = WSAGetLastError();
}
closesocket(s);
return errCode;
}
int main(int argc, char *argv[])
{
auto result = initWinsock();
// check ports
result = createListenSocket(3121);
assert(result == E_NONE);
// cleanup
WSACleanup();
}
When I run this code I find the same failure – the code was unable to successfully bind to port 3121. At least I know the issue is not with the hw_server or my Vivado setup. The issue must lie somewhere with my particular Windows setup. Why is Windows not allowing a process to bind to this port? I did some searches for issues related to blocking and opening of ports. This turns up some useful information on Windows Server and Hyper-V reserving certain port ranges. What Windows calls Administered Port Exclusions. From a Command Prompt you can type the following to list any reserved port ranges: netsh int ip show excludedportrange protocol=tcp.
And there we have it! Something in Windows has added the port ranges I need to the excluded port range. Seems simple enough, there are commands to remove port exclusions. So I try the following: netsh int ip delete excludedportrange protocol=tcp numberofports=1000 startport=3000. I try this command and run the hw_server. And once again found that it is unable to bind to the needed ports.
These are not the ports you are looking for
At this point, I decide that if I can’t figure out how to change the ports that Windows is reserving, maybe I can just convince Vivado to use a different set of ports. In networking, port collisions are not uncommon, and so common convention is to allow client and servers to use different ports. Looking at the command line options for hw_server I find the -s flag which tells the server to use a different port. Precisely what we want! From the command line I try: hw_server.exe -s tcp::50000. And this time the server runs, albeit with a few additional warnings about other port conflicts. But the hw_server process successfully runs.
****** Xilinx hw_server v2020.1
**** Build date : May 27 2020 at 20:24:38
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
INFO: hw_server application started
INFO: Use Ctrl-C to exit hw_server application
INFO: To connect to this hw_server instance use url: TCP:localhost:50000
Warning: Cannot create '3000:arm' GDB server: An attempt was made to access a socket in a way forbidden by its access permissions
Warning: Cannot create '3001:arm64' GDB server: An attempt was made to access a socket in a way forbidden by its access permissions
Warning: Cannot create '3002:microblaze' GDB server: An attempt was made to access a socket in a way forbidden by its access permissions
Warning: Cannot create '3003:microblaze64' GDB server: An attempt was made to access a socket in a way forbidden by its access permissionsUnfortunately, Vivado does not appear to provide a user interface for changing the ports that the Hardware Manager and associated processes expect to use. However, Vivado does provide the ability to run commands manually from a command prompt. It is actually a TCL prompt that will pass non-TCL commands along to the command prompt. So I try to manually connect to the hw_server using port 50000 via: connect_hw_server -url TCP:localhost:50000.
This sadly fails in a different way due to the inability to connect to something called cs_server which I find out later possibly represents the ChipScope server.
INFO: [Labtools 27-2285] Connecting to hw_server url TCP:localhost:50000
INFO: [Labtools 27-3415] Connecting to cs_server url TCP:localhost:3042
INFO: [Labtools 27-3417] Launching cs_server...
INFO: [Labtools 27-2221] Launch Output:
Traceback (most recent call last):
File "git_cs_server\chipscope\server\main.py", line 401, in <module>
File "git_cs_server\chipscope\server\main.py", line 397, in main
File "site-packages\click-7.0-py3.7.egg\click\core.py", line 764, in __call__
File "site-packages\click-7.0-py3.7.egg\click\core.py", line 717, in main
File "site-packages\click-7.0-py3.7.egg\click\core.py", line 956, in invoke
File "site-packages\click-7.0-py3.7.egg\click\core.py", line 555, in invoke
File "git_cs_server\chipscope\server\main.py", line 279, in start_server
File "git_cs_server\chipscope\server\main.py", line 259, in start_server
File "git_cs_server\chipscope\server\main.py", line 293, in __init__
OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions
[22024] Failed to execute script main
ERROR: [Labtools 27-3733] Error during cs_server initialization: Unable to connect to cs_server at URL: TCP:localhost:3042.
connect_hw_server: Time (s): cpu = 00:00:00 ; elapsed = 00:00:14 . Memory (MB): peak = 1042.156 ; gain = 0.000
ERROR: [Common 17-39] 'connect_hw_server' failed due to earlier errors.But do I need Hyper-V?
Another quick look at the command line options for hw_server shows that I can specify ports for the GDB server via the -g and -p options. And a look at the cs_server executable shows that I should be able to set the port that it listens to as well. But, I can see that this is all starting to become quite complex. And even if it works, I really don’t want the toolchain that I use for FPGA development to involve complex scripts to deal with these port exclusions. So I take a step back and ask myself, could this whole problem be fixed simply by disabling Hyper-V? If so it would be a simple solution and so certainly seems worth a try.
First of all, Hyper-V is great. It allows you to host virtual machines like the Linux Subsystem for Windows. Or other versions of Windows like Insider builds. Having a virtual machine setup is certainly nice. But can I live without it in order to get my FPGA development underway?
I can live without Hyper-V
The answer for now is yes. My main use of Hyper-V had been for Windows Phone 10 development. Who does Windows Phone 10 development you say? Well, I do. You see, I am the author of the Tesla Control app available in the Windows 10 Store. As a Universal Windows App, Tesla Control runs on the Desktop PC, Tablets, Windows Phone, and Xbox One. Microsoft’s Windows Phone 10 Emulators run as virtual machines under Hyper-V. And the ability to test your app on many different Windows Phone configurations is certainly important.
However, since Windows Phone 10 devices are no longer being sold, the number of mobile customers for the app continue to shrink over time. And I do have some physical Windows Phone devices that I can still use for testing. So I decide that I no longer have a compelling need for the Windows Phone emulators. Therefore no current requirement for Hyper-V.
So I disabled Hyper-V in Settings | Apps & Features | Programs and Features | Windows Features. And with that Vivado now works as expected. I am able to finally connect to the Basys 3 and program the device. Which means that now I can get back on track with learning and sharing!
Final thoughts
This was quite an unexpected excursion. I did find a blog post that spoke to the port exclusion problems and the existence of an undocumented registry key that should fix the issue. I sure wish I had found that article a little earlier! And no doubt in hindsight it will be easy to see what I might have done to shorten this story. But sometimes you find the learning is worth the journey.
In solving this problem I made good use of both the Digilent and Xilinx forums. Everyone there was as helpful as they could be, but no one was specifically familiar with the Administrative Port Exclusion problem. It would be nice if Microsoft would provide more clarity on Hyper-V and how to manage port exclusions. And I would like to see Vivado offer a user interface for managing and remapping the port ranges used by their tools.
Did I miss something? Please let me know. If you are a Hyper-V user and know how to properly fix this problem please leave a comment!