Lab 8 Details for MPCS 56600

Each lab will consist of a small problem and details of how to proceed. Each lab is intended to give every student hands-on experience with the core concepts and technologies covered during the course.  A student may concentrate, as a team member, on one technology over another for the final project, but labs are designed to give each and every student exposure to all the technologies that come into play.  You need to submit labs to the TAs for grading--see submission instructions below.  Generally, unless otherwise specified, you will have one week to complete each assigned lab.

See the syllabus for information on grading.  Turning in lab assignments on time is required, without exception, and all late deliveries will be penalized, regardless of cause.  Submit your assignments to the subversion repository according to the directions on the syllabus page.

You may write these solutions in any programming language of your choice.  Our suggestion is now is not the time to learn a new programming language along with the concepts themselves.  So our suggestion is to use whatever programming language you  know best.

Lab 8   Due: 4:00 pm, Tuesday, August 6, 2022

You are to complete Problem 1 AND Problem 2 OR Problem 3 (but not both 2 and 3).

Problem 1:  Docker Introduction (5 out of 10 points): 

BACKGROUND:

Like all programming problems, learning a new technology is not an exercise in reading but rather an exercise in thinking and typing.  This lab is designed to give you  hands-on experience in some fundamental skills involved in Docker containerization.  You will generally find the References section below helpful in addition to the required and recommended reading.  When we talk about "Docker", we are talking specifically about the Stable Community Edition of Docker, which is the version we will be using in this class.  The Stable Community Edition provides the basic container engine and built-in orchestration, networking, and security.

In this lab, you will practice creating Docker images and containers. 

WHAT YOU NEED TO DO:

There are two shell scripts that you will use to document your work with this lab (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  Contact the TAs if you're working on Windows.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab, and will ignore any output that we see that is not part of the lab requirements.

Once you run the script, you should see something like:

Script started, output file is typescript

Do NOT accidentally exit out of this shell until you are instructed to at the very bottom of this lab.

Create a working directory (perhaps something like "~/mpcs56600/lab8" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

STEP 1 - Run everything directly on your laptop:

We recommend that you run everything directly on your laptop or desktop.  In general, you will want at least 50 Gig of free disk space, and at least 8 Gig of memory on your laptop, and a minimum of 2 cores, 4 or more being ideal.  If your laptop or workstation cannot satisfy these minimum guidelines, talk to John Hadidian-Baugher and he will be able to give you guidance.  Note that trying to do this work on the Linux Cluster in the MPCS labs may prove less than ideal, as the available diskspace to students is based on a quota.  If you have the space, you may wish to install and run docker and all images and containers directly on your laptop.  Docker is available for most platforms, including MacOS, Windows, and Linux.  This is a relatively easy option and straightforward.  The only downside to this option is that your 51212 development environment is not "contained" and separate from your working laptop environment as would be the case in Option II below.  If you choose to go this route, do the following:

1.  Download and Install docker directly onto your laptop

To install docker directly on a Mac, begin here to install the Stable version.  I would suggest following the instructions on installing the Docker Application natively (aka "Docker for Mac"), as opposed to using the older Docker Toolbox.  You may find a discussion of the differences here.  To install docker directly on Windows 10, begin here to install the Stable version.  Docker for Windows requires 64bit Windows 10 Pro with Hyper-V. Take a look here for more information on requirements for Windows.  If you prefer to install Docker on Linux, you may, as Ubuntu, CentOS, RedHat, etc. are variously supported.  See here for instructions for Ubuntu (but see next sentence!) and here for instructions for RedHat (Fedora).  Further instructions for all platforms can be found here.  NOTE:  If you are installing on ubuntu,  you will find these simple steps much easier:

$ sudo apt-get update
$ sudo apt-get install docker.io
$ sudo service docker status [this will verify that docker has been installed and is running]
$ sudo adduser USERNAME docker [replace "USERNAME" with your own ubuntu username]

Once you have installed docker, run the docker "Hello World" container to verify the sanity of your installation (depending on OS and environment, you may need to manually start Docker (i.e. /Applications/Docker.app, etc.) or even reboot your Windows machine.  Here is an example of what you should see (on MacOS) when you run the Hello World exmaple (you type the command "docker run hello-world"):

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
5b0f327be733: Pull complete
Digest: sha256:1f19634d26995c320618d94e6f29c09c6589d5df3c063287a00e6de8458f8242
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
Visit:   https://cloud.docker.com/
For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

The key is to see that lovely "Hello from Docker!" output (although yours probably won't be in purple!).  Once you see that, congratulations, you're off to the races and may proceed.  Pay no attention to the "Unable to find image" warning, that's normal and par for the course, because that's the first time you ran hello-world and no image was local in your local repository, so it was downloaded from the docker hub.

STEP 2:

Create a personal docker hub ID and repository

The next step is to get real...that is, join the docker community.  Go here and sign up on the Docker Hub and create your own Docker ID.  Signing up on the hub will give you space on the docker hub where you can store your own working containers, and leverage more than 100,000 containers and images that other developers have already created (hello-world is one). 

STEP 3:

General intro to images and containers with a few examples

Make sure you have your runscript working before continuing.

For this first lab, we are going to simply run a few commands and make sure that they work, which will also give you a taste of working within Docker.  First, you will need to document your exercises, and to do this, you will use a Unix program called script.  There are two shell scripts that you will use (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  If you're running Windows, your only option will be to either (a) start a new cmd window, perform all your actions in that window, and then, when finished, select all and then copy and paste the window output into a file and submit that file, or (b) install cygwin and run the script command in cygwin.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab and subsequent labs, and will ignore any output that we see that is not part of the lab requirements.

First, make sure docker is running either on your laptop or in your VM (if you are using one).  If you are working in a VM, do the following commands inside your Ubuntu VirtualBox VM.

Create a working directory (perhaps something like "~/mpcs56600/lab8" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

Ok, the first thing you're going to do is type "docker".  Docker will print out it's help file, which gives the various flags and commands you can pass to docker.  This is a fairly robust list of options, but it will give you an idea of the wealth of things Docker can do.  Beside each of the flags and commands, you will see a short description of what each flag and command does.  For example, beside the "exec" command, Docker prints out "Run a command in a running container."  Thus, the "exec" command can be used to run a container.  We will talk about images and containers in class.

Now type "docker exec --help".  Docker will print out the help for the "docker exec" command.  This will list the Usage pattern along with all the options and will again repeat what the command does, i.e., "Run a command in a running container."  You can pass in the --help option with any docker command.  Try getting help on the run command:  type "docker run --help". 

Now let's get to the meat of it.  type "docker images".  When you give the docker images command, it will print out all the Docker images that you have downloaded locally into your repository.  In your case, you will likely have only one image, something like this:

$ docker images
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
hello-world                                  latest              05a3bd381fc2        7 days ago          1.84kB

This shows that in your local repository you have an image called "hello-world" and its Tag is "latest" and its Image ID is 05a3bd381fc2 and the size if 1.84 kilobytes.  We will talk about tags and Image IDs in class. 

Now that we've seen our Docker images, let's look at some containers.  Type "docker ps -a".  This will list all of the instantiated containers you have created (yes, even without your  knowing it).  When I run that command, I see this:

$ docker ps -a
CONTAINER ID        IMAGE                   COMMAND          CREATED              STATUS                          PORTS      NAMES
ef60e564f68e        hello-world             "/hello"         20 seconds ago       Exited (0) 19 seconds ago                  vibrant_benz
7b06f097a37a        hello-world             "/hello"         About a minute ago   Exited (0) About a minute ago              suspicious_yonath
2c377206d2d1        hello-world             "/hello"         18 hours ago         Exited (0) 18 hours ago                    serene_edison
45406c7c504a        hello-world             "/hello"         18 hours ago         Exited (0) 18 hours ago                    youthful_feynman
17342cabc340        hello-world             "/hello"         18 hours ago         Exited (0) 18 hours ago                    thirsty_shaw

This is a listing of all the containers I have executed.  Wow, that's a lot of containers.  Every time I execute the docker run command, I get a new container!  How wonderful.  It even gives my new container an interesting name, such as youthful_feynman.  It also lists the Container ID for each container, the Image that the container is based on ("hello-world"), and when I created it, it's exit status (or if it's still running), any ports the container has bound (we'll see this later), and the container's working Name.  Note that I know that none of the containers is currently running.  I know that because they've all exited.  I can also type "docker ps" by itself.  When I do this, I'm asking Docker to print out all running containers I have.  There are none, so that command wiill not print anything out if I don't have any currently-running containers.  It prints out something like this:

$ docker ps
CONTAINER ID        IMAGE                   COMMAND          CREATED              STATUS                          PORTS      NAME

Well, great.  Let's get a real image down that can actually run for a while, why don't we?  Type the following command:  "docker run -it ubuntu:14.04 /bin/bash".  Now be careful here and notice your NEW PROMPT:

$ docker run -it ubuntu:14.04 /bin/bash
Unable to find image 'ubuntu:14.04' locally
14.04: Pulling from library/ubuntu
bae382666908: Pull complete
29ede3c02ff2: Pull complete
da4e69f33106: Pull complete
8d43e5f5d27f: Pull complete
b0de1abb17d6: Pull complete
Digest: sha256:32c87496025f5cdc5e56b5e5393a4df102b14dd4928f702b2019b18faf1ec48a
Status: Downloaded newer image for ubuntu:14.04
root@0fc3e8b45816:/#

Your prompt is a tad different.  It appears your username is now "root" and your prompt is a root #.  Congratulations.  You are IN the ubuntu container.  Which container?  In the container with the Container ID of 0fc3e8b45816.  You can verify all this by bringing up a separate xterm window and running the following command:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
0fc3e8b45816        ubuntu:14.04        "/bin/bash"         3 minutes ago       Up 3 minutes                            determined_hawking

There's your Container ID listed 0fc3e8b45816.  It's status is "Up 3 minutes".  That means your container is running!  You can play around all you want in your new Ubuntu container.  Play around by typing a few commands at the # prompt, such as ls -la, ps -ef, echo $SHELL, df, date, etc.  Try running "apt-get update" now, which will update your ubuntu (debian) packages to the latest and greatest.  Amazing.  You just modified the contents of your container.  Now type "exit" or press "^d" and you'll be back at your original $ prompt.  Try typing "docker ps" again.  Notice nothing is currently running.  That's because you exited your container, and it stopped running the /bin/bash shell.  Type "docker ps -a" again.

$ docker ps -a
CONTAINER ID        IMAGE            COMMAND          CREATED             STATUS                          PORTS       NAMES
0fc3e8b45816        ubuntu:14.04     "/bin/bash"      10 minutes ago      Exited (0) About a minute ago               determined_hawking
ef60e564f68e        hello-world      "/hello"         2 hours ago         Exited (0) 2 hours ago                      vibrant_benz
7b06f097a37a        hello-world      "/hello"         2 hours ago         Exited (0) 2 hours ago                      suspicious_yonath
2c377206d2d1        hello-world      "/hello"         20 hours ago        Exited (0) 20 hours ago                     serene_edison
45406c7c504a        hello-world      "/hello"         20 hours ago        Exited (0) 20 hours ago                     youthful_feynman
17342cabc340        hello-world      "/hello"         20 hours ago        Exited (0) 20 hours ago                     thirsty_shaw

STEP 3:

Execute the following commands when instructed to do so.  In another terminal window, create a text file called responses.txt into which you will write your responses when asked to do so in each RESPOND section below.  Feel free to copy and paste output from any commands into your response file. 

How do we launch a container and keep it running?  Basically there are two ways:

pass the -it flags to either docker run or docker exec and launch a shell (/bin/bash):

EXECUTE:  docker run -it ubuntu:14.04 /bin/bash

Now, exit your container by either typing exit or pressing Ctrl-d:

EXECUTE:  exit [or press ^d]

RESPOND:  What did that docker run command you just executed above do?
   
Now, run the image and pass it the –id flag (daemon interactive):

EXECUTE:  docker ps

EXECUTE:  docker run -id ubuntu:14.04

EXECUTE:  docker ps

RESPOND:  What do you see and what did you learn from this?

EXECUTE:  docker exec -it f62568d02b31 /bin/bash
[of course you will use the appropriate container id (or name) from the docker ps command for YOUR container id...NOT f62568d02b31 above.  You can use docker ps [-a] to find this id]

Now, exit out of your container by pressing either ^d or typing "exit".

Ok, great, but notice every time we run a docker images, we seem to be “collecting” containers (you can see them with "docker ps -a")....wouldn’t it be nice to be able to save and work in a single container?  We can.  First, let’s stop our Ubuntu container:

EXECUTE:  docker stop f62568d02b31
[of course you will use the appropriate container id from the docker ps command for YOUR container id...
wait for it, may take a few seconds to shut down...]

Let's now erase all the ubuntu:14.04 containers we've amassed so far:

EXECUTE:  docker rm $(docker ps -a |grep 14.04|awk '{print $1}')

EXECUTE:  docker ps -a

RESPOND:  What just happened?  What did the docker rm command do?

EXECUTE:  docker run -id --name [youruserid]_ubuntu ubuntu:14.04
[supply your own usesrID above in the command]

RESPOND:  What just happened?  Is your userid image currently running in a container or not?  How do find out if it's running?

EXECUTE:  docker exec -it [youruserid]_ubuntu /bin/bash

Ok, great, now we’re in our container.  Now what?  Well, let’s update the Ubuntu operating system with the latest release:

[execute these commands INSIDE your Ubuntu container...]

EXECUTE: uname -a   

EXECUTE:  apt-get update   

RESPOND:  What just happened?
   
Now download the vi editor:

EXECUTE:  apt-get install -y vim

EXECUTE:  which vim

EXECUTE:  cd /; ls -la; whoami

RESPOND:  What did you learn from executing those commands?

Now, exit your container by either typing exit or pressing Ctrl-d:

EXECUTE:  exit [or press ^d]

You have now exited your container and are back at your OS prompt.

EXECUTE:  docker stats -a --no-stream

RESPOND:  What is all this info?  What does it mean?

EXECUTE:  docker stop [youruserid]_ubuntu

EXECUTE:  docker ps

RESPOND:  What does docker stop do (be specific about the signal)?

Now, start your container again:

docker start [youruserid]_ubuntu

EXECUTE:  docker kill [youruserid]_ubuntu

RESPOND:  What just happened?  Be specific about signals.  (run "docker kill --help" to get more info).

Now share a directory from your hard drive with your container (cd to /tmp)

On your host system (or inside you virtualbox VM, if you're running docker inside virtual box), cd to your ~/mpcs56600/lab8 directory.  Now create a new file called lab8.txt in your lab8 directory and edit its contents to be "I love docker".

EXECUTE:  docker run -it -v ~/lab8:/clab8 ubuntu:14.04 /bin/bash

Now, from within your container:

EXECUTE:  cat /clab8/lab8.txt

EXECUTE:  ls –la /clab8

RESPOND:  Discuss what just happened.  Be specific about the docker run command above and the effect of the -v flag...

Now, exit and remove the container you just created...whatever its name or id.
       
Now, suppose we’d like to copy our ~/mpcs56600/lab8/lab8.txt file into your [youruserid]_ubuntu container.  How would you do that?

EXECUTE:  docker cp ~/mpcs56600/lab8/lab8.txt [youruserid]_ubuntu:/clab8.txt

RESPOND:  Is your [youruserid]_ubuntu running?  Does it matter to the docker cp command whether the container is actually running?

EXECUTE (if your ubuntu is not running):  docker start [youruserid]_ubuntu

EXECUTE:  docker exec -it [youruserid]_ubuntu /bin/bash

Now, inside your container:

EXECUTE:  ls -la /clab8.txt

RESPOND:  What do you see?

EXECUTE:  cat /clab8.txt

RESPOND:  What do you see?

EXECUTE:  vi /clab8.txt

Now modify its contents in some way.

Now exit your container.

EXECUTE:  docker cp [youruserid]_ubuntu:/clab8.txt ./modified.txt

cat ./modified.txt

RESPOND:  What do you see?

Now make this all permanent by saving a new image based on your modified container.  That way, you can use [youruserid]_ubuntu for all sorts of new purposes:

EXECUTE:  docker images
EXECUTE:  docker commit [youruserid]_ubuntu ubuntu:[youruserid]_update
ESXECUTE: docker images

RESPOND:  What just happened?  What is the difference between the outputs of the two docker images commands above?

EXECUTE:  docker inspect ubuntu:[youruserid]_update

RESPOND:  What is all this information, in summary?

One last thing to do.  Let's practice saving a container image to a tarball so we can share our images and containers with others.

If you need to (run docker images to find out), get down busybox again:

EXECUTE:  docker run busybox echo Hello 56600 Class Peeps!

Now, let’s copy that image to a tarball (REPOSITORY and TAG are separated by colons in docker commands):

EXECUTE:  docker stop [youruserid]_ubuntu

EXECUTE:  docker save -o ubuntu_[youruserid]_update.tar ubuntu:[youruserid]_update

EXECUTE:  docker rmi ubuntu:[youruserid]_update

EXECUTE:  docker images

Now, let's load that saved image into a :

EXECUTE:  docker load  < ubuntu_[youruserid]_update.tar

Now, let’s see our newly imported image:

EXECUTE:  docker images

Now, run our new image to instantiate a container:

EXECUTE:  docker run -it ubuntu:[youruserid]_update echo Hello 56600 Class Peeps!

Finally, remove the new container you just created with docker rm.

 Make sure you SAVE YOUR SCRIPT FILE. 

Simply type "exit" and you will see something like:

Script done, output file is typescript
Saving output to file: mark.Tue.Jul.24.12-55-01.CDT.2018.out
Problem 2 OR Problem 3:

CHOOSE BETWEEN DOING PROBLEM 2 or PROBLEM 3 (DO NOT DO BOTH)

These solutions are to be writing in the programming language prescribed by the instructions.  The reason for this is simple.  One cannot write a Java RMI solution in Haskell.  That said, you will not need to be "programming" in Java (or Python or C++ or Go) in this lab, although you will be following instructions to build Docker containers and you will run executables in these languages in their respective containers.  The goal of this lab is to get you introduced to the networking capabilities of Docker and introduced to Java RMI (Problem 2) and Google gRPC (Problem 3), so that (in Lab 9) you can distribute your blockchain across multiple containers and mine from multiple containers that are all communicating in a peer-to-peer fashion.

You are to do either Problem 2 or Problem 3.  Either is worth 5 out of 10 points for this lab.  Your key decision point on whether to opt for Problem 2 or Problem 3 will be whether you are working in Java and would prefer the simplicity of Java RMI network communication (Problem 2) or you wish to explore Google gRPC (Problem 3).  Note that you can still do gRPC using Java as well (and instructions are below)...so this is totally up to you.  Everyone working in a language other than Java will likely opt for Problem 3 (although there are no rules here...you can still choose Problem 2 even if  you're programming in Haskell or Go....).

Problem 2:  Docker RMI Introduction: 

BACKGROUND:

Like all programming problems, learning a new technology is not an exercise in reading but rather an exercise in thinking and typing.  This lab is designed to give you  hands-on experience in some fundamental skills involved in Docker containerization.  You will generally find the References section below helpful in addition to the required and recommended reading.  When we talk about "Docker", we are talking specifically about the Stable Community Edition of Docker, which is the version we will be using in this class.  The Stable Community Edition provides the basic container engine and built-in orchestration, networking, and security.

In this lab, you will practice networking Docker containers and synchronous "RPC" mechanisms for container-to-container intercommunication. 

WHAT YOU NEED TO DO:

There are two shell scripts that you will use to document your work with this lab (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  Contact the TA if you're working on Windows.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab and subsequent labs, and will ignore any output that we see that is not part of the lab requirements.

Create a working directory (perhaps something like "~/mpcs56600/lab7" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

STEP 1:

For this lab problem, we are going to code a little container that will accept a synchronous call over RMI to a server.  That RMI Server will echo back the string sent from the RMI Client, and return it to the RMI Client running in a separate container. 

First, make sure docker is running either on your laptop (Option I from the first lab) or in your VM (Option II from the first lab).

STEP 2:

Get Java RMI working in a Docker Container (SERVER)

First, get down an ubuntu container.  Do this by executing the following command (make sure you understand the meaning and significance of -p 8080:8080):

docker run -di -P --hostname rmi_server --name RMIServer ubuntu:14.04

Once you have that container down (and running), obtain a bash shell in that container by running docker exec into the container (you should know how to do this by now).  But to remind ourselves one more time, that would be:

docker exec -it RMIServer /bin/bash

Once you are in the server container, install some software into the container.  You will update the ubuntu OS, then install the vi editor, the Java 1.7.x JDK (via default-jdk), and some net-tools, which will allow you to run network-related utilies including ifconfig.

apt-get update
apt-get install -y vim
apt-get install -y default-jdk
apt-get install -y net-tools
apt-get install -y openssh-client
apt-get install -y unzip
apt-get install -y procps

Inside your server container, create a subdirectory called "/src/lab7/RMI" (explore "mkdir -p").  Next, download ~mark/pub/56600/src.labs/LAB.7/RMI.tgz into your container and untar the contents into your RMI subdirectory.  You can scp it from the Linux cluster directly into your container (as you have installed the openssh-client). 

Once untared, cat the README file and compile the java RMI client and server.  Then execute the RMI server in your container's terminal window and then exec another bash terminal into your same RMI server container and execute the client, to make sure everything is running adequately so far.  Refer again to the README file on how to run the client and server.   You can just use "localhost" for the <server host address> on the client command line, as for this initial test you are running both the client and server inside the same container, ie., RMIServer.

This container will server as your "Server" container for the synchronous exercise of this lab. 

STEP 3:

Get Java RMI working in a separate Docker Container (CLIENT)

Now, create another ubuntu container:

docker run -di -P --hostname rmi_client --name RMIClient --link RMIServer:rmi_server ubuntu:14.04

Once you have that container down (and running), obtain a bash shell into that container by running docker exec (you should know how to do this by now).  Once you are in the client container, install some software into the container.  You will update the ubuntu OS, then install the vi editor, the Java 1.7.x JDK (via default-jdk), and some net-tools, which will allow you to run network-related utilies including ifconfig.

apt-get update
apt-get install -y vim
apt-get install -y default-jdk
apt-get install -y net-tools
apt-get install -y openssh-client
apt-get install -y unzip
apt-get install -y procps

Inside your client container, create a subdirectory called "/src/lab7/RMI".  Next, download ~mark/pub/56600/src/RMI.tgz into your container and untar the contents into your RMI subdirectory.  You can scp it from the Linux cluster directly into your container (as you have installed the openssh-client). 

Once untared, cat the README file and compile the java RMI client and server (javac *.java). 

Next, inside the RMIClient, run "cat /etc/hosts" and note the ip address that Docker assigned for your "rmi_server" in the container.  This magic happened because when you ran your client, you linked your RMIServer container to your RMIClient container (via --link RMIServer:rmi_server) in the docker run call above.  This created an entry in /etc/hosts mapping the IP Address of your RMIServer to the HOSTS alias "rmi_server".

Now, it's time to take this baby for a spin.  Change to your "/src/lab7/RMI" directory, and (with the RMI Server still running in your RMIServer), execute the RMI Client:

root@rmi_client:/src/lab7/RMI# java RMIClient rmi_server 12345 hi
sending hi to rmi_server:12345

root@rmi_server:/src/lab7/RMI# java RMIServer
using address=rmi_server/172.17.0.7,port=12345
hi

You are done with Problem 1. Now with all your containers from this problem running (start them again if you've stopped them), execute the following from a prompt on your host system (NOT in a container): 
$ docker ps                      >Problem1.out
$ docker diff RMIServer         >>Problem1.out
$ docker diff RMIClient         >>Problem1.out
You can shut things down now (Ctrl-C out of java RMIServer).Once back at your host prompt, type:

docker stop RMIServer
[be patient...may take a few secs]

docker stop RMIClient
[be patient...may take a few secs]
This concludes Problem 2.  If you chose to do Problem 2, you are done and ready to submit this lab.

Problem 3:  gRPC and Docker Networking: 

BACKGROUND:

Like all programming problems, learning a new technology is not an exercise in reading but rather an exercise in thinking and typing.  This lab is designed to give you  hands-on experience in some fundamental skills involved in Docker containerization.  You will generally find the References section below helpful in addition to the required and recommended reading.  When we talk about "Docker", we are talking specifically about the Stable Community Edition of Docker, which is the version we will be using in this class.  The Stable Community Edition provides the basic container engine and built-in orchestration, networking, and security.

In this lab, you will practice networking Docker containers. 

WHAT YOU NEED TO DO:

There are two shell scripts that you will use to document your work with this lab (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  Contact the TA if you're working on Windows.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab and subsequent labs, and will ignore any output that we see that is not part of the lab requirements.

Create a working directory (perhaps something like "~/mpcs56600/lab7" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

We will be working with containers that contains Google Protocol Buffers and example gRPC code (from google) for the following language bindings:

java [https://grpc.io/docs/tutorials/basic/java.html]
C++ [https://grpc.io/docs/tutorials/basic/c.html#generating-client-and-server-code]
go [https://grpc.io/docs/quickstart/go.html#install-protocol-buffers-v3]
python [https://grpc.io/docs/tutorials/basic/python.html]

There is a quickstart guide for each language (implementing simple HelloWorld):

gRPC Guides:  https://grpc.io/docs/guides/
gRPC Tutorials:  https://grpc.io/docs/tutorials/
C++:  https://grpc.io/docs/quickstart/cpp.html
Java:  https://grpc.io/docs/quickstart/java.html
Go:  https://grpc.io/docs/quickstart/go.html
Python:  https://grpc.io/docs/quickstart/python.html

These are the simplest introductions to the various language bindings.

STEP 1:

Download the file :  grpc-lab7.tar.bz2. from /home/mark/pub/56600/src.labs/LAB.7  It's over a gig in size, so it will take a minute or so.

bunzip2 it.  (install bunzip2 if you need to).

Load it into docker:

$ docker load < grpc-lab7.tar
18f9b4e2e1bc: Loading layer [==================================================>]  129.3MB/129.3MB
a021c4ee5b3a: Loading layer [==================================================>]  281.1MB/281.1MB
7656f8e9f9d5: Loading layer [==================================================>]  182.3MB/182.3MB
ecb7d1e409dd: Loading layer [==================================================>]  664.6MB/664.6MB
cb19c93edb8e: Loading layer [==================================================>]    208MB/208MB
f6ba9235b10c: Loading layer [==================================================>]  1.886GB/1.886GB
680f7dcf7877: Loading layer [==================================================>]  691.2kB/691.2kB
bcab93a01dd1: Loading layer [==================================================>]  234.4MB/234.4MB
Loaded image: grpc-lab7:mpcs56600

Now, run:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
grpc-lab7           mpcs56600           d1e7bbaa08a1        16 hours ago        3.53GB

Now, run a new docker image to create a GRPC_SERVER container:

docker run -it --hostname grpc_server --name GRPC_SERVER grpc-lab7:mpcs56600 /bin/bash

this creates a new container based on grpc:mpcs5660 called GRPC_SERVER:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED                  STATUS              PORTS               NAMES
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         Less than a second ago   Up 3 seconds                            GRPC_SERVER

Now, exec into your new GRPC_SERVER container:

$ docker exec -it GRPC_SERVER /bin/bash
root@grpc_server:/#

Run the C++ server:
cd to /src/grpc/examples/cpp/helloworld

root@grpc_server:/src/grpc/examples/cpp/helloworld# ./greeter_server
Server listening on 172.17.0.3:50051

Now, confirm that the server is functioning by opening another terminal window, exec'ing into the same container (as above), and changing into the same helloworld cpp directory, and executing:

root@grpc_server:/src/grpc/examples/cpp/helloworld# ./greeter_client
Greeter received: Hello world

You are looking to see Hello world printed out.  Great.

Now, run a python test:

$ docker run -it --hostname grpc_client_python --name GRPC_CLIENT_PYTHON --link GRPC_SERVER:grpc_server grpc-lab7:mpcs56600 /bin/bash

If you open another terminal window on your host, and run docker ps, you should see your new python container:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         41 seconds ago      Up 52 seconds                           GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         10 minutes ago      Up 10 minutes                           GRPC_SERVER

Now, execute the following cat command:

root@grpc_client_python:/# cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.2    grpc_server grpc_server GRPC_SERVER
172.17.0.3    grpc_client_python

Notice that docker has kindly added a HOSTS link to your GRPC_SERVER which is aliased as "grpc_server".  This is nice and very kind of Docker, and the magic happened because you added the  --link GRPC_SERVER:grpc_server line in your docker run command for your GRPC_CLIENT_PYTHON above.

Well, let's see if this stuff works.  Execute:

root@grpc_client_python:/src/grpc/examples/python/helloworld# python greeter_client.py
Greeter client received: Hello you

root@grpc_client_python:/#

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
5098b903220e        grpc-lab7:mpcs56600      "/bin/bash"         8 minutes ago       Up 10 minutes                           GRPC_CLIENT_PYTHON
a0cad78cfbf2        grpc-lab7:mpcs56600      "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_GO
3a597e796815        grpc-lab7:mpcs56600      "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_JAVA
c0fe154dbc9a        grpc-lab7:mpcs56600      "/bin/bash"         2 hours ago         Up 2 hours                              GRPC_SERVER

In your python container you just ran, cd to the /src/grpc/examples/python/helloworld directory, and then execute:

root@grpc_client_python:/src/grpc/examples/python/helloworld# python greeter_client.py
Greeter client received: Hello you

This is wonderful!  The python client (you should of course examine the python client code) has done this to communicate to the GRPC_SERVER in the other container:


  channel = grpc.insecure_channel('grpc_server:50051')
  stub = helloworld_pb2_grpc.GreeterStub(channel)
  response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
  print("Greeter client received: " + response.message)

It has created a channel to the grpc_server listening on port 50051, and has created an RPC stub, and has called the server's SayHello function (remember the server is written in C++) passing it the parameter "you".  The response back from the server is:  "Hello you".  You can of course change the name from the anonymous "you" to your own name.

Ok, so far, so good.  Now let's run a Go container.  First, exit out of the GRPC_CLIENT_PYTHON container.  Then, execute:

docker run -it --hostname grpc_client_go --name GRPC_CLIENT_GO --link GRPC_SERVER:grpc_server grpc-lab7:mpcs56600 /bin/bash

et voila!  There you are in your brand new GRPC_CLIENT_GO container:

root@grpc_client_go:/#

From that other free terminal, execute:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
cb1c098fbbf9        grpc-lab7:mpcs56600   "/bin/bash"         54 seconds ago      Up About a minute                       GRPC_CLIENT_GO
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         25 minutes ago      Up 25 minutes                           GRPC_SERVER

Whoa Nellie!  What happened to the Python container????  Well, when you exited it's bash shell, the container stopped.  No matter, it's still there:

$ docker ps -a
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS                     PORTS               NAMES
cb1c098fbbf9        grpc-lab7:mpcs56600   "/bin/bash"         2 minutes ago       Up 2 minutes                                   GRPC_CLIENT_GO
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         16 minutes ago      Exited (0) 2 minutes ago                       GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         26 minutes ago      Up 26 minutes                                  GRPC_SERVER

If you ever want to start it again, all you have to do is execute:

docker start GRPC_CLIENT_PYTHON in a terminal window and you're back in it.

$ docker start GRPC_CLIENT_PYTHON
GRPC_CLIENT_PYTHON

From another window:
$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
cb1c098fbbf9        grpc-lab7:mpcs56600   "/bin/bash"         3 minutes ago       Up 3 minutes                            GRPC_CLIENT_GO
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         18 minutes ago      Up 6 seconds                            GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         27 minutes ago      Up 28 minutes                           GRPC_SERVER

Then, in another free terminal window you can just exec into the running python container:

$ docker exec -it GRPC_CLIENT_PYTHON /bin/bash
root@grpc_client_python:/#

Ok, back to our Go container.  Get back into the window where you executed the docker run command for GRPC_CLIENT_GO.  It will have the prompt "root@grpc_client_go:/#".

Now, let's execute the Go client against our C++ server.  First, in the Go container, type once more:

root@grpc_client_go:/# cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.2    grpc_server grpc_server GRPC_SERVER
172.17.0.3    grpc_client_go

There she is!  grpc_server again!  Excellent. 

Now, let's run the Go client against the C++ server:

Make sure we have our GOPATH set appropriately (should be...):
env |grep go
See:
PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOPATH=/root/go

If you see your PATH includes /usr/local/go/bin and your GOPATH set, you're "good to go".

cd to the go source directory:
cd ~/go/src

Now, do an "ls" on that directory, and you'll see several directories, including google.golang.org/

That directory contains the go language examples.

Execute:
ls google.golang.org/grpc/examples/helloworld

Here, you'll see the greeter code.  There's a little change we need to make.  cd to the google.golang.org/grpc/examples/helloworld/greeter_client directory:

root@grpc_client_go:~/go/src $ cd google.golang.org/grpc/examples/helloworld/greeter_client

Then, change this code in main.go:

const (
        address     = "localhost:50051"
        defaultName = "world"
)

to this:

const (
        /*address     = "localhost:50051"*/
        address     = "grpc_server:50051"
        defaultName = "world"
)

This way our go program will be hitting our C++ server in the other container.

Now build the go client:
go generate google.golang.org/grpc/examples/helloworld/greeter_client

If all goes well (this is Unix of course), you'll see nothing.  That's good news.

Now, you can execute the go client:

root@grpc_client_go:~/go/src/google.golang.org/grpc/examples/helloworld/greeter_client# go run main.go
2018/07/23 14:14:52 Greeting: Hello world

Hot diggity dog.  It worked.  Pro.

Now, we'll run a Java client.  Exit out of your Go container, and then execute:

docker run -it --hostname grpc_client_java --name GRPC_CLIENT_JAVA --link GRPC_SERVER:grpc_server grpc-lab7:mpcs56600 /bin/bash
root@grpc_client_java:/#

root@grpc_client_java:/# cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.2    grpc_server grpc_server GRPC_SERVER
172.17.0.3    grpc_client_java

Note the actual IP address for the grpc_server (above it is "172.17.0.2").  You'll need that in a minute.

Now, we've got to go into the Java client and modify the server address (similar to what we did in our Go example above). 

First cd to ~/java/grpc-java/examples and then cd to the src/main/java/io/grpc/examples/helloworld/ directory.

Edit the file:  HelloWorldClient.java

In the main() method, we need to update the server IP address with the IP address for the grpc_server from the /etc/hosts file, so modify this line:

    HelloWorldClient client = new HelloWorldClient("localhost", 50051);

to this:

    //HelloWorldClient client = new HelloWorldClient("localhost", 50051);
    HelloWorldClient client = new HelloWorldClient("172.17.0.2", 50051);

Of course we're assuming your grpc_server address is "172.17.0.2".  You'll of course change it to whatever YOUR server's IP address is.

Now, cd back into your ~/java/grpc-java/examples directory, and rebuild the Java code:

Now, rebuild java client:
./gradlew installDist
Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reused, use --status for details

BUILD SUCCESSFUL in 12s
12 actionable tasks: 12 up-to-date

You're looking for that happy green BUILD SUCCESSFUL line.

Now, run the java client:
./build/install/examples/bin/hello-world-client

You should see something like this output from the java client:

root@grpc_client_java:~/java/grpc-java/examples# ./build/install/examples/bin/hello-world-client
Jul 23, 2018 2:40:01 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Jul 23, 2018 2:40:02 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world

Whatever the date, it's that last "INFO" line that's important.  That's the greeting from the C++ server.

Feel free to examine the Java source code that is generated in the following files and directories:
./src/main/java/io/grpc/examples/helloworld/*
./build/generated/source/proto/main/grpc/io/grpc/examples/helloworld/GreeterGrpc.java

This latter file is the java stub genereated by the protocol compiler.

Now with all your containers from this problem running (start them again if you've stopped them), execute the following from a prompt on your host system (NOT in a container):

$ docker ps >Problem2.out
$ docker diff GRPC_SERVER >>Problem2.out
$ docker diff GRPC_CLIENT_PYTHON>>Problem2.out
$ docker diff GRPC_CLIENT_JAVA >>Problem2.out
$ docker diff GRPC_CLIENT_GO >>Problem2.out

Then, submit Problem2.out as described below under Submitting.

Exit out of your Java container. 

You can now stop your C++ server and exit out of the GRPC_SERVER container.

Ensure you have your terminal logs and submit these. You may need to type exit to stop one or more runscript sessions.

REMEMBER:  At any point now you can simply start and stop your containers, so, for example, if you wanted to restart your Java container, and then stop it again,  you'd just execute:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up 45 minutes                           GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_SERVER

$ docker start GRPC_CLIENT_JAVA
GRPC_CLIENT_JAVA

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
61095b85983b        grpc-lab7:mpcs56600   "/bin/bash"         21 minutes ago      Up 5 seconds                            GRPC_CLIENT_JAVA
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_SERVER

$ docker stop GRPC_CLIENT_JAVA
GRPC_CLIENT_JAVA

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_SERVER

Same with any of the other containers.

This concludes Problem 3.  If you chose to do Problem 3, you are done and ready to submit this lab.

References:

You may find the following references helpful (in addition to the links from previous labs):

General Docker Tutorial Links

Docker Cheat Sheet
Learn Docker in 12 Minutes Youtube
Demystifying Docker Youtube
TutorialsPoint:  Docker Tutorial for Absolute Beginners
Docker Overview
Ubuntu package commands

Submission:

For Labs:

Each student should submit their LabN files and any supporting materials to the repo. Please include a README text file that contains any instructions or additional details for the TAs to assist with grading.

Setup:

When you open the invitation URL in a browser tab, you will have to complete the following steps:

Github may ask you to grant permission for GitHub Classroom to have access to your GitHub account. If you are asked this question, you should say yes and grant access to Github Classroom.

You must click “Accept this assignment” or your repository will not actually be created. Do not skip this step!

If you run into any issues, please ask for help on Ed.