Printable View
@rockco , thanks, keep us updated on your lag issues. I find it interesting it has "gone" away now that I kicked everything about. So if it reoccurs, I will kick again. If not, it might have to do with using Memcached vs Xcache. Seems like we are hopefully narrowing down what is going on.
Now that the forum seems to be playing nice, the request is back to have upload image in private messages so users can send proof of mailing without uploading photos showing addresses to other sites
Yes! Thanks so much for sending me the image on how to SWITCH TO CODING mode.Quote:
Originally Posted by FallenPixels
@dataiv
Today at 5:15pm EDT, I caused my ID to go into light lag mode (means I'm prompted for my credentials only on some activities), but it usually gets worse over time. When the issue occurred at 5:15pm EDT, I received a CloudFlare 520 error (missed the step to screen capture the error).
Prior, I was updating posts with very little content, had no issues.
What occurred August 26, 2017 @ 5:15pm EDT to cause these problems?
I was creating a REPLY to the existing post to add the P&G August coupon information and to be efficient:
- I copied the text from the P&G July post.
- Next, pasted the text into the P&G August reply, I used the VIEW SOURCE CODE mode, by clicking it on and then off a few times. The codes looked OK, so I left the formatting codes intact.
- Next, I made some modifications to the August text
- Then, selected PREVIEW mode
- Forum in hung mode
- Then received CloudFlare 520 error page
- At this point, I could not access the forum, so closed my browser
- Each time I attempted to access forum, just hung.
- After about 30 mins, I was able to access forum, but it was slow. I still could not log in cause my ID was locked out.
- Had to wait till 6:55pm EST until which time I was able to access the forum and log in with no issues.
At 7:11pm, I was successful at creating the REPLY post to include the August P&G coupon listings and images. This time, I pasted plain text with out the formatting codes (ie: [B]). I did use SOURCE CODE On and Off to view. Here's the REPLY posting containing the August P&G.
[.... more information in subsequent post. Logged out due to 520 error.]
[dataiv... as I'm authoring this post, I received a 520 error message. This time, I screen printed the message. Will provide more information in the next post once I log out.]
Adding historical information for my reference about ongoing issues
- January 7, 2017 : Large table post created
- January 7, 2017 to April 10, 2017 : Table post modified approximately 15 times
- April 10, 2017 : Started to experience credential lag issues & other problems
- May 5, 2017 : Prior issues resolved - don't know what fixed issue, but vbprojects & dataiv were involved
- June 3, 2017 : Starting to experience credential lag issues & other problems (probably related to lag)
- August 1, 2017 : Prior issues resolved (along with User ID KelJo) - don't know what fixed issue
- August 10, 2017 : Experience credential lag issues
- August 18, 2017 : Prior issues resolved (dataiv updated us about forum rebuild on server)
- August 21, 2017 : Experience lag issue only after updating massive excel data into the existing table post. Table post has been modified approximately 25+ times since it's creation January 7/17
- August 22, 2017 to August 24, 2017: Test with CloudFlare OFF INSERT: >>CloudFlare off from August 21st until now (2pm EDT on the 24th) <<. Lags continue. Delays in refresh increased. Need 2-3 clicks, instead of 1-clicks to refresh
- August 24, 2017 : dataiv made these mods (1) CloudFlare turned ON @ August 24, 2017 2pm EDT (2) switched the VB datastore cache from memcached to xcache (3) kick/restart all services
- August 24, 2017 2pm EDT to August 26, 2017 5:14pm - No issues.
- August 26, 2017 5:15pm EST - in lag mode due to issues when posting the P&G August edition. Light lag mode continues.
@dataiv
Here's the CloudFlare error message I received while I was authoring my prior post. When I received the message, I used the BACK button to go back to my post, added my note to you and saved the post.
Attachment 334287
@dataiv
(re continued updating from prior post)
The consistent thread that causes the problems is when I capture text and paste data into a post. This time, the quantity of text capture was not a lot compared to the mass data I usually update into a table which had caused the past problem.
· Is the action of capturing text and pasting it into a post causing a problem (credential lag)?
· Or is that the captured text has embedded formatting codes, and the embedded codes are causing the problem?
· Or is it the quantity of capture text that’s causing the problem?
Our current status is:
CloudFlare is ON
VB datastore cache is xcache (from memcache)
User (rockco) is using the feature SOURCE CODE On/Off (I started to use this feature just prior to the recent 520 error)
My current log in status:
User ID is being prompted for credentials on certain functions ie MANAGE ATTACHMENT functions and when I toggle to different sections of the forum. Hasn’t degraded to require log-in for PREVIEWS and LIKES (yet).
Did the error code 520 identify the source of the problem?
Here’s my proposed next steps for debugging:
Kick-start services and determine if that clear-ups the credential lag issue – this will confirm if this has been the task that fixes these one-off user problems (like mine & KelJo). If this corrects the problem, we have a workaround for the problem.
Once my ID does not experience the lag, we test to determine which condition of data is causing the problem by using the same July P&G data, into a new REPLY of the same post.
(1) Paste without embedded codes (by copy from July, pasting, use CODING SOURCE-on to remove all embedded codes, then turn off) and then preview & save
(2) Paste with embedded codes, preview & save
By performing this test, we can eliminate the “imbedded codes” as the culprit. If both cause the lag, it may be the quantity of pasted data with respect to the other features (cache / CloudFlare).
Then we proceed to next step of elimination (but always using the July P&G text data) and during this time, I don't do any other type of mass postings.
Thoughts?
Thanks, it is all interesting information. I restarted the services in question. See if that does make a difference.
I quickly looked into what CloudFlare 520 is, and basically it's a catch-all. I can't find any more detail from that error.
If you paste/update a new thread with this information, does the issue occur? Is it only when editing?
When I have a chance in the next few days, but maybe not this week, I am going to try playing around with a copy of that thread and do what you are doing to see if I can reproduce anything. I do see an increase in CPU usage in the logs around 8pm last night which I imagine is related. How so though, I don't know exactly.
In case restarting services has been what clears it up, I've set this to happen daily at 2am ET.
Did the lag disappear now with the restart?
----------
UPDATE @ 8:47pm: discovered something else... starting at 4:54pm yesterday and going on regularly until 9:30pm yesterday your IP shows in the logs with messages like this many, many times:
[x.x.x.x] reached per client soft connection limit: 30 for [30 to 800] seconds, close connection!
Then others, in particular two Russian IPs over and over for the last 24 hours. But no others... just yours and the two Russian ones. So those are unrelated to you, but do you have any plugins or anything configured in your browser that would establish more connections than usual?
I've blocked the two Russian ones, and would love to whitelist your IP from all these limits (meant to protect the server - 30 connections at once is very abnormal), but your IP changes on a regular basis from looking at the logs.
Oh, and my guess for the error 520 is when the server was randomly closing connections for your IP based on the "too many connections" limit being exceeded.
UPDATE 2 @ 9:03pm: Let's see if the auto-restarting helps. Based on the logs I think it will, and think that this whole process may have just uncovered a bug in the web server software being used in this case... maybe... just maybe. One that only affects a few of you (and I really am not sure what provokes the issue if it is indeed a bug in the software, but it sure does look like it right now). Try for a few days and report back and then if it's the case I can file a bug report. Thanks @rockco for continuing to provide these detailed reports! If this helps at all then I will try and explain. If not, then just a red-herring......
Restarting of services has removed the lag problem.
replies: I never install plug-ins (too prone to exposure on unknown web activity)
Adding historical information for my reference about ongoing issues
- January 7, 2017 : Large table post created
- January 7, 2017 to April 10, 2017 : Table post modified approximately 15 times
- April 10, 2017 : Started to experience credential lag issues & other problems
- May 5, 2017 : Prior issues resolved - don't know what fixed issue, but vbprojects & dataiv were involved
- June 3, 2017 : Starting to experience credential lag issues & other problems (probably related to lag)
- August 1, 2017 : Prior issues resolved (along with User ID KelJo) - don't know what fixed issue
- August 10, 2017 : Experience credential lag issues
- August 18, 2017 : Prior issues resolved (dataiv updated us about forum rebuild on server)
- August 21, 2017 : Experience lag issue only after updating massive excel data into the existing table post. Table post has been modified approximately 25+ times since it's creation January 7/17
- August 22, 2017 to August 24, 2017: Test with CloudFlare OFF INSERT: >>CloudFlare off from August 21st until now (2pm EDT on the 24th) <<. Lags continue. Delays in refresh increased. Need 2-3 clicks, instead of 1-clicks to refresh
- August 24, 2017 : dataiv made these mods (1) CloudFlare turned ON @ August 24, 2017 2pm EDT (2) switched the VB datastore cache from memcached to xcache (3) kick/restart all services
- August 24, 2017 2pm EDT to August 26, 2017 5:14pm - No issues.
- August 26, 2017 5:15pm EDT - in lag mode due to issues when posting the P&G August edition. Light lag mode continues.
- August 27, 2017 11:30pm EDT - lag no longer occurring. dataiv kicked started services @ 8:59pm EDT. Activity scheduled for 2am ET
Is there a scheduled time we should co-ordinate the tests so that you can monitor impact to services? Just to confirm, this is my planned test, unless you prefer I do something different. Pre-requisite: no data lag on my User ID.
Using the July data again, I'm creating a new REPLY to this existing post (and not creating a new post yet)
https://forum.smartcanucks.ca/419765...ine-canada-13/
(1) Paste without embedded codes (by copying from July, paste in Windows text editor to automatically remove all source doing, paste into new reply post, use CODING SOURCE-on to ensure all code removed, then turn off feature, preview & save
.... if lag occurs (stop testing at this point until after services have been kick-started)... if lag does not occur, proceed to next case.
(2) Paste with embedded codes, preview & save
... As long as User ID is not lagging, perform this task
If Case #1 fails, it's not embedded codes causing the problem.
If both cases cause the lag, it may be the quantity of pasted data OR too much content for the ongoing post. Status of these server services are: VB datastore cache is xcache, CloudFlare ON.
Just try whenever convenient and let us know here. I don't think we'll manage to find a good time for both of us :). With a restart happening every 24 hours based on the logs, unless you do whatever causes this a number of times in one 24 hour period, it probably won't occur unless you do it a bunch of times (if this is what it is). But try the above and see, I think it is worthwhile, however, switch the order on a second 24 hour period (after the 2am restart), and maybe just do #1, #2, #1, #2 until you see the issue. Then try #2, #1, #2, #1, etc... until you see the issue. My feeling is it shouldn't take too long and that it will happen regardless of which you do.
Neither scenarios, Case #1(click here) & Case #2 (click here) caused credential lag.
I also went ahead to update MASS data (click here) into the table post, and it ALSO DID NOT cause credential lag today.
Will monitor ID access up to 2am EDT kick start.
When I want to create the error condition, it doesn't happen. Darn, never goes as planned.
@rockco
How has it been with the restart every day at 2am? Just wondering if you've been experiencing the same issue or if that one has gone away.
I've had no lag at all; my perception, the daily restart allows User ID a clean start for the day. Unable to "force" create the lag condition with test data, so now have to abandon that task. Will monitor for potential lag the next time I have to do mass data updating to the existing/new forum post.Quote:
Originally Posted by dataiv
Glad that it is solving the lag issue for you. Will keep that going until there is another resolution as I do now believe this to be a software bug in tracking the # of "active" connections.
Update on attachments posted today: there was a small file corruption issue on the partition that holds the post attachments. As a result, attachments posted from about 3-4PM today until 9PM today likely need to be re-uploaded and won't show until they are. Anything from before then should all be OK.
Sorry about the inconvenience!
