April 24 Technical Journal

Triumphs:
One area of advancement in our project has been our working with Mr. Lee and one of his students to develop a UI (user interface) for our dots and boxes game. Before this, we were playing through the PyCharm terminal so this is a step forward. The first version of the UI is depicted below. Other progress we made was in the integration of our Q-learning work with the Monte Carlo search tree. The Q table would be used instead of random rollout in the Monte Carlo. We also completed the Problem Statement and Background sections for our official report.


Preliminary user interface for the dots and boxes game

Struggles:
Because some people were out due to illness and college visits, some progress was slowed down. Our plan had been to transfer our Q-table into the Monte Carlo Search Tree but formatting differences resulted in incompatibility. As a result, a new Q-table had to be made for the Monte Carlo. The generation of the Q-table (part depicted below) was also an issue as the running of the Q-table generation would not bring much progress. The "Kolo" ratios printed as the games were played fluctuated, improving in the 50/50 direction and then sometimes dropping. Another struggle being encountered is an unclear sense of direction as now that the Q-table is being integrated into the Monte Carlo, we are not sure of what to do next. Robert was out sick for all of last week so that also made things more complicated.

Part of the Q-table generated

Code:
Much of the push these past two weeks has been to get the Q-tables to work with the Monte Carlo code (depicted below). The code Will, Puja, and Edmond were working on (Wozzy) would use the Q-table to generate reward and therefore because MontesQ. The Monte Carlo code stored actions in an array of possible actions and the Q-table had all this condensed into number sets corresponding with boxes and moves. We are unsure as to how to complete integration with this format difference so hopefully our visit to Caltech can answer how to translate the number sets into the Monte Carlo compatible array.

Code on integrating Q-learning into Monte Carlo

Accomplishments and Plans:
For the user interface, we plan on linking it with the game by assigning the coordinate points to the lines users can click on. Our plan going forward is that after meeting at Caltech tomorrow, we will be polishing the game up and I think reintegrating with the code section. Because our learning algorithms are being meshed, I think it would be best if we reconvened back into one group. We also plan on making a presentation to accompany our report and have already begun the basic slides. We also look forward to continue completing sections of our report.

Comments

Post a Comment

Popular posts from this blog

April 10 Technical Journal

Image
Code: Working right now as sort of a middle man between the Q learning and code sides, I was helping out a bit with the code as well since the last technical journal on March 13. While I am still mainly focused on the Q learning side, I worked with Edmond on how to improve user interaction with our game. This would come in the form of changing how the lines are inputted. Our current game requires us to input coordinate points to create lines and our new idea is to designate certain coordinates' pairings as a number (depicted below). For example, the line connecting (0,0) and (1,0) is 1. The pattern would be like a zig zag. Q Learning Algorithm: Continuing on the reinforcement learning look into the Q learning algorithm, Connie and I were working with code we got off of Github which stored Q values in a Q table file. Running it over the period, a statement of "Kolo #,#" was printed. The number started off very unbalanced with the first at 0 and the second at 99. Ove...
Read more

March 13 Technical Journal

Image
Tic Tac Toe Data: We first planned on spending this two weeks looking into datasets for tic tac toe and processing it as a pre-trial for our dots and boxes. What we found and did though, I though was far more insightful. Initially, I was looking around for datasets and found from UVA a dataset for tic tac toe intermediate game states. We thought this could be something as the UCI dataset that we had first come across only gave endgame states. An image of the UVA dataset is posted below. Discussing with Mr. Lee, though, we could not decipher what the non-zero/one numbers represented and decided to move on looking, a sort of dead end. We inferred that -1 and 1 were X and O while 0 was blank. We would eventually later find another way to gather datasets (see Tic Tac Toe Q Learning below). Part of Tic Tac Toe intermediate game states data Tic Tac Toe Monte Carlo: While we had been looking into datasets, it turns out that Will was experimenting with the Monte Carlo search tre...
Read more

Start of School - Sep 12 Journal

Image
Through these several weeks, we have begun our journey to exploring research in machine learning by starting our paper airplane research project, hearing undergraduate research presentations, and learning the foundations of machine learning and the basic principles of research.  Foundations: Our first several weeks covered the foundations of research and more specifically machine learning. In a class activity, all the groups came up with several truths about research and the class was able to collectively develop a set of rules and expectations for the research journey. From "always keep on trying" to "look to nature for inspiration," many of these truths resounded with me as I hope to learn more about this field via research in this course.  The foundations of machine learning were also covered through a series of videos. From explaining neural networks to linear regression, these videos established a baseline from which my group and I could work o...
Read more