In Game Conversation history I had seen a few discussion...

« Back

828 days, 5 hours, 22 minutes ago

qinic

In Game Conversation history I had seen a few discussion...

In Game Conversation history

I had seen a few discussion on the merit of keeping the the status quo when taking over an open position vs. having your own agenda and disregarding anything that the previous player had done with regards to relationships to other players (races) in the game.

A number of players do look at conversation history and try to keep treaties intact.

However it is fairly painful to step through the turns and to look at old messages to find out what went on.

Since I wanted to present and/or have a workshop at the planets CON on reading API data into excel I started writing a little tool to gather conversation intelligence and am probably 70% there.

At the moment I have a spreadsheet with a macro that reads all turndata and extracts all messages into an array.

Now I like to present these messages in a fairly easily digestible format. So the idea is to display each send and received message between any given race in a spreadsheet column making sent messages right bound and received messages left bound similar to sms messages on an Android phone.

I also thought I would want to have all conversations on the same worksheet so you can see all conversations with all players on the same sheet.

But now would I start at row one for each of them and essentially have a race specific timeline or would I want to keep the timelines synced so you can see what conversations with different races happened around the same time. Which would have it's advantages but would result in some huge gaps between messages on exchanges with players that had less conversations.

Just like to hear some thoughts really.

Cheers
Qinic

828 days, 3 hours, 47 minutes ago

RE: In Game Conversation history I had seen a few discussion...

I'd use the first column for Turn number, second for 'Sent By', third for 'Received By' and fourth column for the actual message contents.

If it can be used on Finished games to collect all sides messages into one spot for research, it'd be used far more often then just replacement players.

827 days, 19 hours, 24 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Right, I might have been overthinking this. You can just use filters to see conversations between just 2 players. I will probably add a column for conversation partners so you can filter by just one value rather than specifying receiver and sender combinations. Which will also enable an easy way to get an bar chart of messages between players to get a quick overview of who was chatting to who.

I agree with your finished game usage, at the moment you have to specify your race ID to load turns from that players perspective I will add an option to use all as a parameter to get a full history.

It will take a while to run and use a fair bit of data transfer (~1.5MB per player per turn)

826 days, 11 hours, 54 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Ok Anyone willing to alpha test the spreadsheet??
Be warned though this data gathering macro runs for a while.
Example Giant Melee 120 turns 35 Races = 120 * 35 = 4200 data sets
On my half descent Laptop it takes around 3 seconds per data set to extract the messages that makes in ~3.5h.
I did code it so it will skip Open and Dead races so the actual data set number will be lower but I would still expect it to take at least 1.5 to 2h.

Anyway anyone that like to give it a try send me a PM with an email address to mail it to.

826 days, 11 hours, 15 minutes ago

RE: In Game Conversation history I had seen a few discussion...

> On my half descent Laptop it takes around 3 seconds per data set to extract the messages that makes in ~3.5h.

Would it be faster to do the data extraction in another language, creating a CSV file?

826 days, 9 hours, 27 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

@Whisperer well VBA is not know for it's speed and efficiency so yes absolutely. If I had to guess relatively easy would be a C# implementation, the fasted would be C++ under linux ensuring that the parser can parse the data in parallel using multiple cores, you could probably optimize such that you only limitation is your network bandwidth. You download around 1-4 Gbyte for a full game turnfile load for all players. On my connection I could do that in about 10 minutes.

But the purpose was to showcase using VBA in Excel to read the data as it is convenient to use and could be adopted by a lot more people than the more efficient languages.

Doing this via a Java Script addon would most likely not be much faster, but any of the Java gurus can feel free to prove me wrong

826 days, 8 hours, 57 minutes ago

RE: In Game Conversation history I had seen a few discussion...

This will be a tremendous achievement when complete! I look forward to seeing full published logs of the completed championship games, and agree that it will also benefit the OP's intent of helping get rapidly brought up to speed on a game's history when you take over a vacant race.

826 days, 8 hours, 56 minutes ago

the pale destroyer

RE: In Game Conversation history I had seen a few discussion...

I have a workbook that parses full incremental turn data for my running games with numerous tools running off the complete game data set.
VBA is slow at the best of times... but now I'm exclusively using a mac where Office only utilses one core and the sandboxing means a macscript call for the API it's ludicrous.
Must port it at some point.

826 days, 8 hours, 19 minutes ago

RE: In Game Conversation history I had seen a few discussion...

For any analysis of all player data in finished games, I recommend using the /game/loadall?gameid=XXXXXX API call. It will send down a zip file with everyone's data from each turn in one shot, and should require less data transmission (though the files can still be quite large).

It doesn't currently include the last turn however, so you would still have to do those separately.

826 days, 7 hours, 57 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

@Darth+Balls well it does work, it will give you either all messages in a finished game either from one persons perspective or all, or in any running game it will give you your message history, so if you take over a slot you can see what went on, or you might just like to have a history of your exchanged messages.

@The+Pale+Destroyer cool, like to have a look at that if you don't mind.

@Big+Beefer thanks, I saw that call in the API documentation, when I started my goal was to provide a history for people who jump into a running game, the give me everything from everyone was bolted onto that after @Glyn's suggestion. I will have a look at how that download looks like to see if that can be parsed more efficiently

Anyway if anyone want's to try it out let me know I can email them the sheet

822 days, 8 hours, 33 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Pretty happy with how the sheet works now.
https://drive.google.com/file/d/0B_CRHUQrNwCreDFIbFBjZV9DRnM/view?usp=sharing

822 days, 6 hours, 32 minutes ago

singularity

RE: In Game Conversation history I had seen a few discussion...

@Qinic
I like it a lot. This is something I would use regularly when joining as a replacement, and possibly at end of game to see what was really going on.

However, I could not get the macro to work. I use LibreOffice 5.3.2.2. I turned on Macros and got the error message "BASIC syntax error. If not allowed outside a procedure" when I pressed the button.

Ideally, I'd like this script to run in under 10 minutes, but if it takes more that is not a major issue. It could be run in the background or even overnight.

Some comments to make it more usable:

1) Each body starts with a line similar to this: "The Missing Colonies of Man (PLAYER_NAME)" - which just adds unnecessary text. You already have the sender detailed in Column B.

2) You are formatting CR/LF as "_x000D_".

3) Column G lists all the recipients of the message in one place (great!), but the message itself is repeated x times which is unnecessary. Can you de-duplicate?

Nice work!

822 days, 5 hours, 59 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

@Singularity
Thank you

Done the coding in MS Office 2010, need to get a copy of LibreOffice and see if it contains the libraries that are needed, if it is just a case of cleaning up the code to make it compatible than I shall do that.

I think you struggle to make it run that fast and I am not sure I can as most of the time is spent parsing the JSON which is all done in code that I borrowed from somewhere else (hence the second module)

With regards to the headline, I could easily leave that out as that is the header line bolted onto the body, could easily be skipped.

For the CRLF is use vbCrLf const. I could use CHR codes if you prefer, I am replacing the html line break, not sure if excel handles just LF

I was thinking about the dedub, the thing is that it is how the client handles it. If you send to 3 players you will send 3 messages and the sender stays but receiver changes, I left them in in order to not have gaps if you start to filter on sender and/or receiver. It also sends a copy to self which I did remove, I also remove the Open communication message when you first sent an ambassador as that was just mega annoying.
I agree when you select a party of more than 2 in Collumn G they are a bit annoying. I could make it an option to merge all multicasts into one message and list all receivers, in which case, party filter Collumn G will work great, but receiver filter collumn C would not work so great.

Good feedback, thank you.

822 days, 1 hours, 1 minutes ago

RE: In Game Conversation history I had seen a few discussion...

I opened the sheet in Excel 2013, entered the ID of the game I wanted to view messages from and hit "Get Messages." Then it failed and I debugged the error and discovered that I needed to create a folder C:\temp and give write privileges to the folder. Upon running it again I received Run-time error '6': Overflow. I could create a virtual machine and install MS Office 2010 on it for testing but wondered if anyone else has had trouble testing it with Office 2013.

821 days, 21 hours, 46 minutes ago

RE: In Game Conversation history I had seen a few discussion...

Will there be functionality to sort by multiple threads?
Sometimes there are different threads of discussion, and seeing two threads overlap might be confusing, since you are replying to different messages.

Also, some threads to/from another player will have multiple recipients (a group conversation). There should be a way of marking that.

821 days, 17 hours, 23 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

@Darth+Balls

Sorry should have mentioned that you need a C:\temp folder for the storage of temp data during execution
It works fine in Office 2013 I tested it, could hit debug and let me know which function and line of code fails with the overflow?
In the meanwhile I will see if I can re-create the error by renaming my temp directory, make it fail than fix it and start again, maybe that created a unstable situation.

@Psydev

Would love to do the thread grouping but that data is only available in the latest turn under activities, so you loose that info for the turns before that. If you look at earlier turns in the client you will see that dashboard will stop displaying messages at some point and only the diplomacy message lock shows them, and there it will suffer from the same mash up of messages and the multiple same message issue on multi player conversations.

The group conversations are marked in a way by having more than 2 races mentioned in the party column, if party is greater or equal to 60% of active players in a given turn I marked it as public to prevent ridiculously long party strings.
Also once I add the option to pull multiple copies of the same message content under one entry with multiple receivers option it will be clear from the recipient field.

I looked into LibreOffice and it does not support VBA it supports generic Visual Basic, so it might require a lot of re-coding and from what I read on forums like stackoverflow (my goto forum for all programming needs) it is also not documented very well. So at the moment I don't think I have enough time to port it. If you want to have a go at that I be happy to explain my code to you wherever you get stuck.

821 days, 13 hours, 13 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Thanks for @Darth+Balls for the heads up on the timer error, uploaded a quick fix here:
https://drive.google.com/file/d/0B_CRHUQrNwCrVS1wZGVsN0pBbEk/view?usp=sharing
Other changes still in progress

821 days, 4 hours, 40 minutes ago

RE: In Game Conversation history I had seen a few discussion...

Wow I've got to say I've spent well over an hour today reading all the exchanges in Gemini and even now I've gained some new insights into it. This is a really, really powerful tool and I'm grateful for it. I knew that @Ra and I were verbose, but seriously close to 2000 messages is more than I would have imagined when our next highest competitors exchanged only just over 500 messages. Thanks for easing the stroll down memory lane. :)

820 days, 17 hours, 24 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Thank you @Darth+Balls, glad you like it.
I agree it provides a real insight. The stats are currently double counting as the message that is sent shows up as a received message next turn on the receiver account, on multiple recipients this issue is multiplied.
Currently coding an select-able option to consolidate these into one entry.

You also get sucked into reading a lot I think I have spent half of my development time snooping :)

The tool can be a good ally in deciding your diplomatic moves.

819 days, 19 hours, 42 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

New version uploaded:
https://drive.google.com/file/d/0B_CRHUQrNwCraV9MZkZvQjlNMzA/view?usp=sharing

Change Log:
'Version 1.0: Fixed issue where timer will generate an overflow situation
' Removed header from message text to reduce text length
' Added option to consolidate group messages and send/receive copy into one
' Changed line break to LF from CRLF in message body

Pre-loaded the Aquarius War and also added a histogram of messages sent showing that Emork the Chatterking (sorry Emork could not help myself) overtook Regicide early on in most messages sent.

With Darth Balls input that he sent a ton of messages in the Gemini war it highlights again how important Diplomacy and Communication is in this Game.

Next on my improvement list is to try and see if there is any performance gains to be had as it does take quite some time to run the macro if you grab all data from a finished game.

In the meantime, enjoy the Aquarius War chat or use it to download in the background.

819 days, 14 hours, 34 minutes ago

RE: In Game Conversation history I had seen a few discussion...

I like your newest revision of the script. It makes it a lot easier to track 3 or 4 party communications but in doing so it also dramatically increased the length of the recipient list and made it more cumbersome to select your intended recipients, especially when you go in with a simple intention like "I wonder what the chatter was like between player 4 and player 8." With only 11 or 14 recipients to choose from, it was easier to pick out those two names in the previous script than it was for me when selecting from the approximately 152 recipient combinations that I counted in the Aquarius War. I know that it is impossible to have one's cake and eat it too (sorry if that doesn't translate well for the non-native English speakers) but is there a way that we can improve it somehow to keep the ease of access to the multi-party messages without sacrifice the speed and usability?

The only other thing that I can think of that might fix that dilemma is to see if you can use CONCATENATE or another function to reduce the names of the sender/recipient columns and reduce the size of the filter list so that they don't include double entries for example 3) Bird Man (Darth) and also 3) Bird Man (Dead) or 3) Bird Man (Open). Even that wouldn't dramatically cut down the list from 152 to much lower but it might reduce it by 10 or 15 entries.

Another option could be to have one sheet designed for ease and speed to select two-party communications (simple) and another sheet designed for multi-party communications (advanced).

Brilliant, brilliant, brilliant! This sheet just saves so much time, even after waiting the hour and a half for it to run on my computer, and is leagues beyond logging in and out of the game as different races to get different perspectives and using the time machine to pick through the messages. Thank you so much.

819 days, 14 hours, 28 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Hi Darth Balls, I hear you and yes striking that balance might be difficult, have you tried the party filter on the right for your desired filtering?

819 days, 14 hours, 22 minutes ago

RE: In Game Conversation history I had seen a few discussion...

You mean there are more than just the first 4 columns!?

How would you describe the contents of the Party column? It looks like the list is only about a third as large as the recipient column.

819 days, 14 hours, 12 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Yes there is 7 columns, the last one column G called party does what you ask and only included race and slot number so even if the owner changes it stays consistent. It also ignores sender vs recipient. So let's say Feds talk to lizard than all of their exchanges will be marked Feds(1) Lizard(2) no matter who has sent it of the too, the order is always in ascending slots for this list.
Three way conversations will have 3 members etc.
That should help filtering the messages as you desired, which was the purpose of creating that data column in that way and took a while to figure out.

819 days, 14 hours, 8 minutes ago

RE: In Game Conversation history I had seen a few discussion...

Thanks for the clarification and explanation of the Party Column that makes much more sense. On my monitor I had the resolution such that I thought the last column was the message content column.

Knowing about the Party column I can't see myself ever using the recipient and sender columns to select the thread; why choose it twice when it is so much easier to find in the Party list?

I will be moving the Party column forward ahead of the other two and hiding the Count column on the sheet I save on my desktop unless you can think of an adverse affect that will cause because Count seems to be used only in the backend for statistics.

819 days, 13 hours, 52 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Yes mhh well I use a dual monitor setup with 1920x1200 on each so I did not cater for smaller screens very well.

If you move the columns the next time the script will run it will break, as I programmed it fairly lax. I could in a later edition make that feasible so it figures it out by the header, but than it still breaks when you change these, so now I could lock some aspects of the sheet down and password protect it but I want to also enable other people to change it anyway they like if they have the skills to re-code it at the same time.

You have the option to zoom out or hide fields or resize column width.
I am also open to change the field order in the next version if that makes more sense.

819 days, 13 hours, 46 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Another note on Party column, if a conversation had a lot of members (I set it to 60% or more of active players atm) it marks it up as public, which is not strictly correct but I did not see an easy way of detecting the truly public messages in the mymessages data set. Maybe @Big+Beefer or someone else with this knowledge could chime in and let me know how this can be determined from the limited data in the mymessages array.

819 days, 11 hours, 29 minutes ago

RE: In Game Conversation history I had seen a few discussion...

Am I the only one severely concerned by the fact that conversations from a previous player are accessible AT ALL? I mean - the first time I shared my skype info with an ally to coordinate our turns, I did not exactly expect my (ingame) private messages to show up for everyone to see once the game was over.

In my opinion there should be a clear warning that is always there to remind people that what they are sending will not remain private - and should they wish a message to be private, they should use an out-of-game private message to the player they wish to contact.

Right now, this automated tool to extract all conversation history, together with people using google docs to upload such conversations makes me really worried... I do not wish anyone that I did not give my data to, to link my planets.nu account with my skype account.

While this is not the fault of the conversation tool - I would be very happy to see the functionality to mark messages as "account-bound" (vs. race-bound) so that when a player drops / is dropped from a game, or when the game ends, such messages are permanently deleted.

819 days, 10 hours, 53 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

@Aral, I agree with you in a number of points. I am not sure I like that the messages are open after a game ends or when you take over. It might be better to start with a clean slate and never have the messages disclosed.
I also think that that should be made clearer in the client that all messages will be public knowledge once the game ends and will be available to someone that takes over your slot if you resign/get dropped.
The tool obviously only gives you a convenient alternative to using the game client.
I do think you should only use PMs to exchange personal details but again the client does not warn you about that enough.
With regards to plastering them on google docs, you are probably right I should only upload empty versions, will change that later.

819 days, 10 hours, 11 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

Empty version:
https://drive.google.com/file/d/0B_CRHUQrNwCrNnZHS216d3pxblk/view?usp=sharing

819 days, 9 hours, 47 minutes ago

RE: In Game Conversation history I had seen a few discussion...

@Aral: You are right that there should be a warning... someone might share their phone number, email or home address. I'd even think a warning for the public posts is a good idea since they actually show up in Google searches.

However, nothing else should be changed. The ability to investigate in-game messages helps expose Rule 2 violators... people's posts are like fingerprints... they cannot mask their style. It isn't always obvious, like an American pretending to be a Canadian/British and spelling words wrong (or 'American' spelling to be fair).

In general it is a bad idea to use Private messaging (or email) for in-game communication, unless you don't mind being accused of Rule 2 violations.

BTW, everyone really should divulge who their friends are in their profile... this tool Qinic has made is only going to result in more people discovering obvious pre-game relationships.
___

@Qinic: You are doing a great job, better than anything I would have made for sure.

819 days, 8 hours, 34 minutes ago

RE: In Game Conversation history I had seen a few discussion...

@Qinic: thanks for reflecting upon my concerns!

@Glyn: apart from the warning, I think my idea from above would work: people could flag their messages as "volatile"/"account bound" when they are non game strategy related - the playstyle messages would still remain for history. And if you get "volatile" messages about alliances / game related stuff, you should be wary of that player anyways...

819 days, 5 hours, 38 minutes ago

RE: In Game Conversation history I had seen a few discussion...

@Aral: I've stayed up late hitting refresh and taking screenshots as one by one a handful accounts go from un-viewed, viewed to ready in the same two hour span. But what first drew my attention to those accounts was finished Sectors where they didn't have any in-game messages.

Course I've seen one guy sending phoney messages to his own accounts to avoid suspicion, thankfully he gave wasn't sneaky enough to avoid other tell tale signs.

The problem with your idea is that it makes it too easy to have private messages and even more people wouldn't use the in-game messaging.

I'd like to see in-game messaging notify you via email, and the private messaging via profile page system have a delay built in to discourage being used as an alternative.

If cheating wasn't so widespread problem on this site I wouldn't care... but alas we have well known personalities on this site that are cheating and getting away with it for years.
_ _ _

I'd like to see the in-game messaging system improved for relaying information, making it the best method to use... such as adding planets/ships info that doubles as a link to them and for sending border information or relaying Reports.

819 days, 4 hours, 56 minutes ago

talespin

RE: In Game Conversation history I had seen a few discussion...

You need some other hobbies, Glyn :D

818 days, 20 hours, 6 minutes ago

qinic

RE: In Game Conversation history I had seen a few discussion...

@Glyn, thank you, I am pretty happy how the data presentation turned out. The programming however is cutting corners, but hey it works.