Update on Recent Software Issues
Tuesday, December 23, 2008 - COO
On Saturday, December 20, 2008 we were alerted to a problem with a hand involving PHILHELLMUTH and DOUBLEBALLER.
We realize that this issue is of great concern to UB’ers and the poker community. Since we learned of the problem we have worked tirelessly investigating and fixing the software issue that was responsible for the incorrect payout. We have now released an official statement, which you can read it on our corporate site tokwiro.com.
I became aware of the problem on Saturday evening, shortly before I made my original post on this blog. I must admit that the information on twoplustwo was very helpful to me in getting up to speed right away.
I know many of you wanted information immediately after the incident happened. I did make a post on the UB blog to let everyone know we were aware of the problem and the seriousness of it. At the time I did not comment further because I wanted to make sure that we had all the facts before we made any public statements. I would hate to say something that is inaccurate or even false because I rushed it out before understanding the problem completely.
The problem is truly and honestly just a software malfunction. I do realize this issue caused a great deal of concern because Phil Hellmuth, a spokesperson for the site, benefited from the malfunction and that is quite the coincidence. For this reason we are providing the public with as much information as possible and hope that by doing so, and answering as many of your questions as possible, you will be reassured that our site is a safe place to play.
The Issue:
As we explained in our press release, the incorrect payout was caused by a software malfunction. The software malfunction occurred when the winning player was disconnected from the hand at the precise millisecond the software was determining whom to award the prize. This in conjunction with the “player’s state” data being cleared from memory caused this extremely improbable incident to occur.
Originally we were unable to reproduce the error in our test environment, until we programatically forced a disconnect to happen at the precise millisecond the hand was being awarded.
We have never had an issue like this reported previously and we have reviewed all of our database and application logs for December and November and have been unable to find any historical occurrences. We are continuing to go back further into the historical data to ensure no hands from the past need to be corrected. Analyzing this data takes time but we will continue to go back further and further into our history to ensure no one else was affected.
The second, related issue:
During the weekend we released an update to our game servers which included new business logic to prevent the first incident from ever happening again. The new update unfortunately created a new logic issue resulting in 36 hands with incorrect payouts. The 36 hands affected were lower limit razz, stud, and hold’em hands. Upon learning of this problem, we were able to addressed it quickly and the affected players were refunded immediately. We are extremely confident that everything has been corrected and order has been restored but we’re continuing to monitor the system and are performing extensive tests to ensure there are no incorrect payout or disconnect issues in the future.
An unrelated issue:
Another player made a comment on our blog (Mark) and emailed our support team about a tourney (2120428) where 700 chips were mysteriously deducted from his stack mid tourney. What happened was his chip stack was reset to his starting chip amount (3000 chips). This happened because he won a satellite into the very tourney he was already playing in . This uncovered the unrelated problem, where the system reset his chip stack to the starting amount. This software issue was addressed immediately and Mark’s account was refunded.
Our company takes a lot of pride in our work and i think it’s safe to say we are embarrassed by these issues. We have an amazing software development team who have accomplished so much this year. I assure you they take this very seriously and I’m confident they have dealt with these issues professionally and I do not expect anymore problems to occur.
Our company is committed to being the most transparent site on the net and to providing our players with a personal touch that other sites do not. I speak to many players myself, I grant a lot of interviews, we offer phone support and our security team speaks with players on a daily basis. We hope that you start to notice the improvements we are continually making to improve our service and transparency.
I will try to blog and respond to your comments/questions the best I can over the holidays.
Happy holidays everyone!
Cheers,
Paul Leggett
Related posts:
- Quick Update on Software Issues Hi Everyone, Here is a quick update on the recent...
- December Software Issues We recognize that the community wants more information regarding the...
- Hand #1162170993 Earlier today we learned about a poker hand where our...
- Spring Into Our Software Update We might be a few days ahead of schedule if...
- The Software Fix Is In Our engineers might not have the cure for the winter...
Tags: 2+2, Cereus, Phil Hellmuth, Two Plus Two
December 23rd, 2008 at 8:25 pm
Hi Everyone,
The link to our press release will be ready soon. The content is similar to what I have included here.
Thanks,
December 23rd, 2008 at 8:42 pm
Thanks for your lengthy explanations of the incidents that occurred on Saturday.
The way you explained the tournament chip shortage left me feeling rather unsatisfied. A good number of the players in that tourney got in via satellite, but I didn’t hear of a single other case of stacks being reset.
There was also a mention (on 2+2) that the final table chip count was correct. If that was indeed the case, who got my chips?
Is it perhaps because I won entries both on UB and AP, albeit under different user names? (Note that I played ONLY the UB entry, but that the other entry (which sat out the entire time) managed to finish 24th! Better than I did.
Also, it appears that my stack was reset DURING the first hand in question. How else could I have 2980 chips at the beginning of the second hand?
May I trouble you to be a little more specific in this regard, as the fact of simply winning an entry via satellite was accomplished by a good number of people.
Thanks.
December 23rd, 2008 at 8:45 pm
OH! I think I get it now. One of the (multitude of) satellites I played finished AFTER the main event started. Does that have something to do with the problem?
December 23rd, 2008 at 8:55 pm
Hi Mark,
Yes, that’s what triggered the error in our software as I understand it.
Also, if I recall correctly, there was a blind for 20chips right after your chips were reset, which explained the 2980 chips instead of 3000.
I hope this helps. I am much more informed about the other two incorrect payout problems to be completely honest but I did review your problem with our software developers.
If you have additional concerns I will do my best to educate myself with further information and get back to you.
Sorry for the problems.
Paul
December 23rd, 2008 at 9:07 pm
I have been reading twoplustwo after making this blog post and I came across a question that a poster had, which I thought about as well during the investigation and I agree it needs to be addressed.
‘NoLimitLeagues’, thank you for raising the question.
At this time we are confident that the incorrect payout issue with PHIL’s and DOUBLEBALLER’s hand was not caused by a recent change to the software. It is a simply just a very improbable combination of events that caused it to happen. We are continuing to investigate this further and we are examining historical logs to see if it had occurred in the past.
I will try to comment further in the near future with a better update on this specific question as I learn more.
December 23rd, 2008 at 9:17 pm
Please tell us if the final chipcount in the tourney was correct or the 700 was missing
December 23rd, 2008 at 9:35 pm
I have updated the link to our press release in the above post.
You can access our press release on our corporate website here.
http://www.tokwiro.com/press-releases.asp
If any of you have further questions about recent issues with our software, please post them here or email support and I will do my best to respond in a timely manner.
Paul
December 23rd, 2008 at 9:37 pm
It was kind of crazy to see how many people were all over Hellmuth for this one. Hellmuth is such a massive poker brand, he would have very little to benefit from cheating online and much more to lose. Like he needs to cheat anyways… he’s Phil Hellmuth! 11 bracelets!!!
Nice to see you guys are taking care of issues up front. The commitment shows.
December 23rd, 2008 at 11:08 pm
Hi Paul,
Thanks for the explanation. Just one quick question. Does it show that the “disconnect” was on my end? I am aware that I could experience tiny disconnects that I might not notice, but I wasn’t having any types of connection issues prior to, during, or after this match (I was in a game at another site at the exact same time, no problems there). In fact, I very rarely have connection issues. Just wondering, thanks.
Rob
December 23rd, 2008 at 11:16 pm
Well, I guess it’s progress that Ultimate Bet is just incompetent and not still cheating.
December 23rd, 2008 at 11:34 pm
Paul,
Thanks for the update and thanks also to Dan for the detailed explanation.
I, too, am still curious about the final table chip count and, if it was correct, who got my 700 chips.
Mark
December 24th, 2008 at 3:32 am
I have 2 concerns that keep me from playing your site…
Mark Says:
December 23rd, 2008 at 8:42 pm
Thanks for your lengthy explanations of the incidents that occurred on Saturday.
The way you explained the tournament chip shortage left me feeling rather unsatisfied. A good number of the players in that tourney got in via satellite, but I didn’t hear of a single other case of stacks being reset.
There was also a mention (on 2+2) that the final table chip count was correct. If that was indeed the case, who got my chips?
Is it perhaps because I won entries both on UB and AP, albeit under different user names? (Note that I played ONLY the UB entry, but that the other entry (which sat out the entire time) managed to finish 24th! Better than I did.
Also, it appears that my stack was reset DURING the first hand in question. How else could I have 2980 chips at the beginning of the second hand?
May I trouble you to be a little more specific in this regard, as the fact of simply winning an entry via satellite was accomplished by a good number of people.
Thanks.
——————————————————–
As he says he won 2 entries (UB and AP) using two different user names and is quick to add (only played one). Deja Vue… No doubt he sees both hands if logged on from 2 computers sitting side-by-side. Not saying Mark did, but he occupied 2 spots in this tourney. Illegal as hell on any honest site.
Sounds like this combining of two serperate poker sites has created another opportunity for cheaters. I get the distinct impression that Mark feels he or an accomplice could have played both. How many teams are working these 2 sites and how large are they? Scary as hell.
And as to the most recent “software glitch”. If this disconnect had happened seconds before, Phil would have won the hand and no amount of complaining would have changed the outcome. Only the showing of hands and immediate outside pressure even got this looked at. The only solution other than fixing buggy software is to institute disconnect protection for all tourneys and at least the medium and large regular games. I don’t see you doing that, either, as that system is quickly abused and a huge processor drain. To be honest, the old saying that a fool and his money are soon parted seems to really apply to UB and AP. Good luck to the fools who go where even angels fear to tread.
December 24th, 2008 at 7:50 am
Where is Phil’s apology?
December 24th, 2008 at 12:27 pm
Cutthroatman,
It is not possible for the two accounts owned by the same player (1 on AP, 1 on UB) to be logged in at the same time. Only ONE can be played at a given time. It doesn’t matter if simultaneous logins from the same IP are involved or if attempted from different IP’s.
It simply cannot be done.
FWIW, before the merge into Cereus, I had the same player moniker (iykisa) on both sites. The first time I logged into AP after the merger, the system required me to change my moniker there since it recognized that the name was in use at UB.
Partly because of the situation in the tourney involved, I don’t expect to play my AP account in the future – only my UB account.
Mark
December 24th, 2008 at 12:45 pm
Two questions.
Why should anyone believe a word you say?
Why would anyone ever play at your site when there are so many trustworthy alternatives available?
December 24th, 2008 at 1:04 pm
I posted my letter to your COO in my blog. I have cashed out but not be cause of this little bit. Although, it does relate to the reason. The elimination of all-in protection was done in a very slipshod fashion. When one gets a disconnect, they are removed from the table. The reason given seems possible until we note that the person remained at the table. That is a remote possibility but suspect. I been dumped from your tables on various streets. I don’t see the outcome and I cannot access the hand history do to that happening. It appears that was not the case with the hand as I’ve heard it explained.
December 24th, 2008 at 5:34 pm
Hey Mark,
I just took a quick look at the tournament you were playing and it looks like the total is off by roughly 700 chips. I am having someone from our poker room and a DBA review the hand and provide me with a detailed analysis. When I get it, I will share it with you.
All the best,
Paul
December 24th, 2008 at 6:12 pm
Hi ‘Why Play at UB?’
Regarding your two questions.
1. I am providing our players and the poker community with as much information as possible. If you are interested in me personally…you can learn more about me by looking at the many interviews I have done which you can find online, a video interview is available on our corporate website http://www.tokwiro.com and you can ask about me in the community.
Also, we are working on new ideas for how we can improve the transparency into our security so we can reassure the public that our sites are safe. We should be making announcements about this in the new year.
2. I believe people play at our site because they have a had a good experience with us. Whether that was a good experience with winning or interacting with our customer service. A lot of people play with us because they like our promotions or cashier, etc. Also, a lot of the biggest names in poker play on our site. A lot of these players came back after they saw how we handled the cheating scandal that we inherited. Specifically, how we fought hard to get their money back and refund millions of dollars.
I realize that many poker players have used their power as a consumer to deny us business. This is everyone’s right and it doesn’t upset me. It is our job to rebuild trust overtime. We have our work cut our for us but we are committed to being transparent and the most accessible/personal poker site on the net.
We hope that as time goes by you will continue to see the positive effect of the changes we’ve made to this organization and even if you chose to never play with us again, hopefully one day you might say “those guys aren’t all bad” and we’ll feel we’ve succeeded.
I hope that helps.
Paul
December 24th, 2008 at 6:48 pm
Sorry Mr. Leggett, but the fact that with millions of hands dealt each day that within the 37 instances of this problem, one involved your highest profile player (sorry Annie).
First, the probability of this happening to Phil must have been miniscule. Also – given the history with high profile players have “superuser” accounts – you can understand everyone the apprehension that everyone has with this explanation.
Second – would you be willing to publish the hands of the other players affected by this glitch? If you are truly in “full disclosure” mode – then I think that the good people at pocketfives and twoplustwo would be willing to investigate each hand in question and possibly contact those involved in those hands to verify your validity that all pots have been paid correctly. Since the internet people originally found the problem that UB originally denied existed – I think that having those same people investigate this would only be right.
Finally – I have closed my account on UB. I have not been an active player for a while on your site (for various reasons). But I felt obligated to close my account because I no longer felt that UB/AP/Cereus was committed to running a fair game.
December 24th, 2008 at 7:27 pm
Hi BobbyC,
I completely understand the apprehension.
The odds of this happening on a hand where one of our pro’s was playing, and that pro actually benefiting from that malfunction, are long at best.
I know Nat Arem well, and I was hoping to consult with him after the holidays to see if he could help me with a plan on how UB can share more information with the public in order to help satisfy the concerns you and others have.
If you email our poker security department pokersecurity@ultimatebet.com and ask for me, I will try and involve you in this discussion. I am not opposed to sharing these hand histories. However, I can only share information that will not jeopardize the privacy of our players but I think we can work around that.
Also, just to make sure I am clear about this, so far we have found no additional historical occurrences of the malfunction that occurred with Phil’s hand. But we are still have a lot of data to analyze, so it is possible that it has happened in the past. I will provide an update as we progress with our analysis but it’s a lot of data and takes some time. The other 35 hands involve a seperate problem with our software that we introduced when we enhanced the logic in our software to prevent an incorrect payout like Phil’s from reoccurring.
I am sorry that you closed your account and I hope that one day you and others will regain some confidence in our poker room.
Paul
December 24th, 2008 at 9:16 pm
You said there we no other instances of the bug but here is proof . You say you checked December
Stage #1136089263: Holdem No Limit $0.50 – 2008-12-02 03:48:48 (ET)
Table: TUSTIN (Real Money) Seat #6 is the dealer
Seat 6 – OZMANZ69 ($175.34 in chips)
Seat 1 – SBOM333 ($86.20 in chips)
Seat 2 – MIKE DACIUK ($100 in chips)
Seat 3 – MINES13W ($24.20 in chips)
Seat 4 – ROYALSTARS ($88.11 in chips)
Seat 5 – LUCKY_SPACE ($68.50 in chips)
SBOM333 – Posts small blind $0.25
MIKE DACIUK – Posts big blind $0.50
*** POCKET CARDS ***
Dealt to LUCKY_SPACE [Ad 9d]
MINES13W – Folds
ROYALSTARS – Folds
LUCKY_SPACE – Raises $1.75 to $1.75
OZMANZ69 – Folds
SBOM333 – Calls $1.50
MIKE DACIUK – Folds (Timeout)
*** FLOP *** [3c 7d 2d]
SBOM333 – Checks
LUCKY_SPACE – Bets $3
SBOM333 – Folds
LUCKY_SPACE – returned ($3) : not called
*** SHOW DOWN ***
LUCKY_SPACE – Does not show
MIKE DACIUK Collects $3.80 from main pot
*** SUMMARY ***
Total Pot($4) | Rake ($0.20)
Board [3c 7d 2d]
Seat 1: SBOM333 (small blind) Folded on the FLOP
Seat 2: MIKE DACIUK (big blind) Folded on the POCKET CARDS
Seat 3: MINES13W Folded on the POCKET CARDS
Seat 4: ROYALSTARS Folded on the POCKET CARDS
Seat 5: LUCKY_SPACE HI: [Does not show]
Seat 6: OZMANZ69 (dealer) Folded on the POCKET CARDS
December 24th, 2008 at 11:44 pm
Paul,
I’d like to ask for your help in another matter – unanswered emails to your support department. In some cases, I have resent them FOUR times and clearly marked them “FOURTH REQUEST”.
Here are the issues:
1. Frequently the “Prize Info” button is greyed out on tourney lobbies, thus disabling it.
2. Poker tourney lobbies do not update unless they have the focus (i.e., you must click on the lobby window).
3. A Player didn’t bet the nuts on the river. Hand #58509296-122 at 12:40amPLOHL-001 (Pot Limit tournament Omaha 8/b). Possible collusion?
4. There is no option for switching avatar visiblity on and off.
Thanks for your attention.
Mark
December 28th, 2008 at 5:50 am
Paul,
You’ve said (above):
“We have never had an issue like this reported previously and we have reviewed all of our database and application logs for December and November and have been unable to find any historical occurrences. ”
But there is a hand history from which contradicts this which you must be aware of by now. I just wanted to bring it to light to anyone who may be reading this here.
Is this hand history genuine? Do you concede it’s a previous example of this “glitch”? Why wasn’t it picked up in your analysis of all hands from the last two months? If you missed it, how do you expect us to have confidence in anything else you say on the matter?
hand history in question follows:
Stage #1136089263: Holdem No Limit $0.50 – 2008-12-02 03:48:48 (ET)
Table: TUSTIN (Real Money) Seat #6 is the dealer
Seat 6 – OZMANZ69 ($175.34 in chips)
Seat 1 – SBOM333 ($86.20 in chips)
Seat 2 – MIKE DACIUK ($100 in chips)
Seat 3 – MINES13W ($24.20 in chips)
Seat 4 – ROYALSTARS ($88.11 in chips)
Seat 5 – LUCKY_SPACE ($68.50 in chips)
SBOM333 – Posts small blind $0.25
MIKE DACIUK – Posts big blind $0.50
*** POCKET CARDS ***
Dealt to LUCKY_SPACE [Ad 9d]
MINES13W – Folds
ROYALSTARS – Folds
LUCKY_SPACE – Raises $1.75 to $1.75
OZMANZ69 – Folds
SBOM333 – Calls $1.50
MIKE DACIUK – Folds (Timeout)
*** FLOP *** [3c 7d 2d]
SBOM333 – Checks
LUCKY_SPACE – Bets $3
SBOM333 – Folds
LUCKY_SPACE – returned ($3) : not called
*** SHOW DOWN ***
LUCKY_SPACE – Does not show
MIKE DACIUK Collects $3.80 from main pot
*** SUMMARY ***
Total Pot($4) | Rake ($0.20)
Board [3c 7d 2d]
Seat 1: SBOM333 (small blind) Folded on the FLOP
Seat 2: MIKE DACIUK (big blind) Folded on the POCKET CARDS
Seat 3: MINES13W Folded on the POCKET CARDS
Seat 4: ROYALSTARS Folded on the POCKET CARDS
Seat 5: LUCKY_SPACE HI: [Does not show]
Seat 6: OZMANZ69 (dealer) Folded on the POCKET CARDS
December 29th, 2008 at 5:40 pm
Wow, now Hellmuth is saying that he’s played 100 hands online (presumably many of them at UB?) where the pot was awarded to the wrong person.
(He says this on Rounders radio. Look up the twoplustwo thread about this whole scandal for a link to the audio in case you think I’m making this up)
Paul, can you please comment on this, because it’s obviously at odds with your claim that the hand that caused all the controversy is the first occurance on your site, and it seems utterly bizarre that Hellmuth would claim this now but would not have mentioned it to anyone at UB support before.
The idea that he’s had the pot incorrectly shipped to him 50 times before online but nobody has ever kicked up a fuss about it before is also implausible. Looking forward to when you resume commenting on this issue.
January 9th, 2009 at 11:21 am
Hi Gary and MarkR,
Paul has made a formal post surrounding your inquiry to hand #1136089263. We appreciate your patience while we looked into this issue as we wanted to give a full, detailed response.
http://blog.ultimatebet.com/2009/01/december-software-issues/
January 9th, 2009 at 12:17 pm
Hi Everyone,
Regarding Phil’s comments… I believe he is probably referring to hands where there was a software malfunction and not the pot being awarded incorrectly.
The Dec 2008 malfunctions are the only incorrect payout issues that I have heard of since I joined the company.
Paul
January 9th, 2009 at 12:27 pm
Some of you who post comments have probably noticed that we are currently monitoring/approving them.
I apologize for the delays that occur sometimes in approving and responding to them. I instructed our Admin to hold off on approving a lot of your comments until I could write a more detailed report, which I posted here yesterday.
http://blog.ultimatebet.com/2009/01/december-software-issues/
We are trying to respond to all comments as fast as possible. However, sometimes we prefer to get all of the information and respond in a more complete way.
I hope that the post from yesterday answers all of your questions regarding the incident and we will do our best to keep up with your questions.
Thank you for your patience while we completed our investigation.