tag:blogger.com,1999:blog-27488238.post115998462001347056..comments2024-03-22T11:34:45.165+01:00Comments on taw's blog: Why Perl Is a Great Language for Concurrent Programmingtawhttp://www.blogger.com/profile/16972845140253292628noreply@blogger.comBlogger30125tag:blogger.com,1999:blog-27488238.post-29116149635704028612018-01-27T20:05:57.881+01:002018-01-27T20:05:57.881+01:00The complaint about slow INSERT in SQLite is so co...The complaint about slow INSERT in SQLite is so common a problem it's even got a FAQ entry:<br /><br /> https://sqlite.org/faq.html#q19<br /><br />Spoiler: wrap the INSERTS in a transaction.<br /><br />SQLite isn't being "braindead," it's giving you the ACID compliance you expected when you chose to use a proper database.Warren Younghttp://tangentsoft.comnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-6412444097662344892016-12-10T02:54:01.516+01:002016-12-10T02:54:01.516+01:00"Now what's so great about Perl concurren..."Now what's so great about Perl concurrency ? It works. Access to all Perl libraries."<br /><br />Far from it. You only have access to thread-safe librairies. Is LWP thread-safe? Its documentation doesn't bother telling the user about it. And the same is true for most librairies.<br /><br />And the fact that a library is written in pure Perl is not enough to make it thread-safe. Even builtin Perl functions can be thread-unsafe, like rand() or readdir() (see "Thread-Safety of System Libraries" in perlthrtut).<br /><br />Look at LWP's dependencies on CPAN. A lot of external modules (MIME::Base64, File::Temp, IO::Uncompress::Inflate, Digest::MD5, IO::Socket, etc.). Are they all thread-safe? Their documentation says nothing about it (at least for the few examples I just gave).<br /><br />So does it really work? I guess it does, if your program is very small and simple. Otherwise, the answer is probably no.Anonymoushttps://www.blogger.com/profile/14155115799694927233noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-65141932196537223992009-04-20T18:03:00.000+02:002009-04-20T18:03:00.000+02:00AnyEvent::HTTP is the most promising thing I have ...AnyEvent::HTTP is the most promising thing I have found having looked high and low and posted here for help:<br /><br />http://perlmonks.org/?node_id=758739Terrence Brannonhttps://www.blogger.com/profile/02568302240846756317noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-24917358141774498512007-10-17T00:08:00.000+02:002007-10-17T00:08:00.000+02:00suprised nobody mentioned POE. it's an event-drive...suprised nobody mentioned <A HREF="http://poe.perl.org/" REL="nofollow">POE</A>. it's an event-driven multitasking environment for perl. there's an example of a parallel link-checker in POE here: <A HREF="http://www.stonehenge.com/merlyn/LinuxMag/col41.html" REL="nofollow">http://www.stonehenge.com/merlyn/LinuxMag/col41.html</A>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1695220939448292292007-08-05T12:37:00.000+02:002007-08-05T12:37:00.000+02:00Jon: No really, sorry.Jon: No really, sorry.tawhttps://www.blogger.com/profile/16972845140253292628noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-83683745965791192442007-08-05T12:34:00.000+02:002007-08-05T12:34:00.000+02:00I wondered if you'd had any more thoughts about th...I wondered if you'd had any more thoughts about this?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-58140441070514219472007-07-19T02:07:00.000+02:002007-07-19T02:07:00.000+02:00You mean how are the correlations done? I have a s...You mean how are the correlations done? I have a script functioning for that. You can see it on: http://psychonomics.com/stockscan2.html<BR/><BR/>It's not fully functional - but if you use any sector with an x in it at the begining of the symbol eg XBD, XAL and choose a low enough threshold and the periods you want, you should get a table out with correlations. Datafeed must be set to yahoo.<BR/><BR/>As you'll see, it's very slow - precisely because it's doing a sequential process. And we always know in advance how many correlations it has to do - and the script tells you on the output page. So theoretically if it has to do 66 pairwise correlations, we would need 66 parallel processes... I guess.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-77918129835446629732007-07-18T21:04:00.000+02:002007-07-18T21:04:00.000+02:00Jon: How do you run the script to produce 3 correl...Jon: How do you run the script to produce 3 correlations ? It's probably easy to automate.tawhttps://www.blogger.com/profile/16972845140253292628noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-64530819134456619062007-07-17T17:00:00.000+02:002007-07-17T17:00:00.000+02:00I think that is probably a useful way to look at i...I think that is probably a useful way to look at it. And no doubt I have a throuhput problem. The question is how to solve it?<BR/><BR/>To work out one correlation, the script can do very quickly. But to do 2 correlations, the script has to be used twice, and 3 correlations, 3 times and so on. So how do you make all this happen in parallel? Would one solution perhaps be to make the script (or the necessary component) reproduce itself? So that if you need 3 correlations done, you'd have 3 scripts.<BR/><BR/>Would that be a possibility?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-82245735916447761092007-07-15T21:04:00.000+02:002007-07-15T21:04:00.000+02:00Jon: Threads and other kinds of parallelism are on...Jon: Threads and other kinds of parallelism are only useful for latency-bound problems, not for throughput-bound problems.<BR/><BR/>In latency-bound problems program spends a lot of time waiting for something and doing nothing - for example it asks one websites, waits, then asks another, waits, and so on. Doing such things in parallel can make it much faster because many things can be done during the wait, and it doesn't cost you anything to wait for more than one thing at the same time.<BR/><BR/>In throughput-bound problems programs works as fast as it can and it simply cannot do things any faster. Making such problems parallel won't affect the total running time at all, and it may even become slower.<BR/><BR/>Most problems are somewhere in between these two. Yours sound more like a throughput-bound problem, so I don't think parallelism is going to help you much.tawhttps://www.blogger.com/profile/16972845140253292628noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-65179290545876140662007-07-15T20:56:00.000+02:002007-07-15T20:56:00.000+02:00Hi I came across your blog and believe it might ha...Hi<BR/> <BR/>I came across your blog and believe it might have a bearing on a problem I'm having. Basically I have an output table produced from a perl script that contains the results of a formula in each cell - a correlation in fact. However each cell is produced sequentially and this takes a lot of time - especially when hundreds of correlations are run. And the mathematical procedure required for each cell is identical. A method of producing the result for each cell in parallel would be much faster, I believe. I'm not a programmer myself, so any thoughts on how to go about this - or find someone who could help with the problem - would be appreciated. <BR/> <BR/>Thnx<BR/> <BR/>JonAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-74835694193437078492007-04-08T22:38:00.000+02:002007-04-08T22:38:00.000+02:00For everyone else, that wants to use threads for "...For everyone else, that wants to use threads for "concurrent programming", the following two lines are mandatory:<BR/><BR/>use threads;<BR/>use threads::shared;Unknownhttps://www.blogger.com/profile/10711875458922509233noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-75637398673134958742007-04-08T14:58:00.000+02:002007-04-08T14:58:00.000+02:00Hi Taw,Read your blog with great interest as I hav...Hi Taw,<BR/><BR/>Read your blog with great interest as I have failed to get Perl threads running and has started to look at Erlang as an altenative.<BR/><BR/>I have experienced the same problem with slow SQLite updating and at that occasion I was using class-dbi-sqlite. The solution to increase speed when I was creating a database with data was to turn off "autoupdate"<BR/>"FFF::group::member->autoupdate(0);"<BR/><BR/>When I had finished creating all records, I turned on autoupdate and everything was saved as expected. "...->autoupdate(1)"<BR/><BR/>With that small change in my code SQLite worked like a charm.<BR/><BR/>Now I will get Perl threads running and shelve the idea with Erlang, as there is not even SQLite for Erlang.<BR/><BR/>Thanks<BR/><BR/>StefanUnknownhttps://www.blogger.com/profile/10711875458922509233noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-69307215286130535042007-04-08T14:53:00.000+02:002007-04-08T14:53:00.000+02:00This comment has been removed by the author.Unknownhttps://www.blogger.com/profile/10711875458922509233noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-52449890806438984442007-04-05T17:58:00.000+02:002007-04-05T17:58:00.000+02:00it works amazingly, thx so much!!it works amazingly, thx so much!!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-84596996586052861612007-04-05T14:05:00.000+02:002007-04-05T14:05:00.000+02:00ketvin: You need to create the threads first, and ...ketvin: You need to create the threads first, and only join them later:<BR/><BR/>my $thr0 = threads->create('one');<BR/><BR/># main + $thr0 are running now<BR/><BR/>my $thr1 = threads->create('two');<BR/><BR/># main + $thr0 + $thr1 are running now<BR/><BR/>my $thr2 = threads->create('three');<BR/><BR/># main + $thr0 + $thr1 + $thr2 are running now<BR/><BR/>my $result0 = $thr0->join();<BR/><BR/>my $result1 = $thr1->join();<BR/><BR/>my $result2 = $thr2->join();tawhttps://www.blogger.com/profile/16972845140253292628noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-20940613186216060102007-04-05T10:04:00.000+02:002007-04-05T10:04:00.000+02:00Your codes are very cool, give me a lot of inspira...Your codes are very cool, give me a lot of inspiration.<BR/><BR/>Question, I don't really understand how you call your thread concurrently. <BR/><BR/>For example, if i am having a few threads, each doing different thing, how do i able to make it run concurrently?<BR/><BR/>eg.<BR/><BR/>my $thr0 = threads->create('one');<BR/>my $result0 = $thr0->join();<BR/><BR/>my $thr1 = threads->create('two');<BR/>my $result1 = $thr1->join();<BR/><BR/>my $thr2 = threads->create('three');<BR/>my $result2 = $thr2->join();<BR/><BR/>and i have different sub of one,two and three, the way i call th join() will just make it run in serial. <BR/><BR/>Just asking for opinionAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-75341110146375309122007-01-29T21:03:00.000+01:002007-01-29T21:03:00.000+01:00Saying "Perl is great at ..." is the same as sayin...Saying "Perl is great at ..." is the same as saying "I don't know much about ...."Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1160160298440998462006-10-06T20:44:00.000+02:002006-10-06T20:44:00.000+02:00#!/usr/bin/env rubyrequire 'open-uri' require 'thr...#!/usr/bin/env ruby<BR/><BR/>require 'open-uri' <BR/>require 'thread'<BR/><BR/>def http_status l<BR/> begin<BR/> open(l) do |f|<BR/> f.status.join(' ')<BR/> end<BR/> rescue OpenURI::HTTPError => e<BR/> e.message<BR/> rescue => e<BR/> "ERROR -- #{e.message}"<BR/> end<BR/>end<BR/><BR/>def deq1 q<BR/> thr, l, m = q.pop<BR/> thr.join<BR/> puts "#{m}: #{l}"<BR/>end<BR/><BR/>q = Queue.new<BR/>nthr = 0<BR/><BR/>ARGF.each do |l|<BR/> next unless l =~ /^(http:\/\/.*)$/<BR/> Thread.new(l) do |l|<BR/> q << [Thread.current, l, http_status(l)]<BR/> end<BR/> if nthr >= 20<BR/> deq1 q<BR/> else<BR/> nthr += 1<BR/> end<BR/>end<BR/>nthr.times { deq1 q }Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1160159847352440462006-10-06T20:37:00.000+02:002006-10-06T20:37:00.000+02:00#!/usr/bin/env rubyrequire 'open-uri' require 'thr...#!/usr/bin/env ruby<BR/><BR/>require 'open-uri' <BR/>require 'thread'<BR/><BR/>def http_status l<BR/> begin<BR/> open(l) do |f|<BR/> f.status.join(' ')<BR/> end<BR/> rescue OpenURI::HTTPError => e<BR/> e.message<BR/> rescue => e<BR/> "ERROR -- #{e.message}"<BR/> end<BR/>end<BR/><BR/>def deq1 q<BR/> thr, l, m = q.pop<BR/> thr.join<BR/> puts "#{m}: #{l}"<BR/>end<BR/><BR/>q = Queue.new<BR/>nthr = 0<BR/><BR/>ARGF.each do |l|<BR/> next unless l =~ /^(http:\/\/.*)$/<BR/> Thread.new(l) do |l|<BR/> q << [Thread.current, l, http_status(l)]<BR/> end<BR/> if nthr >= 20<BR/> deq1 q<BR/> else<BR/> nthr += 1<BR/> end<BR/>end<BR/>nthr.times { deq1 q }Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1160140950074318172006-10-06T15:22:00.000+02:002006-10-06T15:22:00.000+02:00Quickshot: I don't threads in Perl 6 are even impl...Quickshot: I don't threads in Perl 6 are even implemented. Or LWP. Perl 6 is pre-alpha software right now.tawhttps://www.blogger.com/profile/16972845140253292628noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1160139644972233452006-10-06T15:00:00.000+02:002006-10-06T15:00:00.000+02:00Just as a small question, but does it work under p...Just as a small question, but does it work under perl 6 as well? Or does that seriously change the problem?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1160084455156002262006-10-05T23:40:00.000+02:002006-10-05T23:40:00.000+02:00like i said earlier it was just a quick proof of c...like i said earlier it was just a quick proof of concept<BR/><BR/>heres one that doesnt slurp it all into memory, and the result cache isnt needed if you prefilter the url list with sort | uniq... if you really wanted it you could add a third state variable to the loop or throw it in an ets table, and to reduce memory of strings you can pack them into a binary with <<>> which brings it back down to 1 byte per char<BR/><BR/><BR/><BR/>-module(checker).<BR/>-compile(export_all).<BR/><BR/>start() -><BR/> {ok, Fd} = file:open("urls.txt", [read]),<BR/> Pending = enqueue(20, Fd),<BR/> master_loop(Fd, Pending),<BR/> file:close(Fd).<BR/> <BR/>master_loop(_, []) -><BR/> done;<BR/>master_loop(Fd, Pending) -><BR/> receive <BR/> {http, {RequestId, Result}} -><BR/> {value, {_R, Url}} = lists:keysearch(RequestId, 1, Pending),<BR/> {{_, Status, _}, _, _} = Result,<BR/> io:format("~s ~b~n", [Url, Status]),<BR/> NewIds = enqueue(1, Fd),<BR/> Merged = [ {Id, U} || {Id, U} <- Pending, Id /= RequestId ] ++ NewIds,<BR/> master_loop(Fd, Merged);<BR/> M -><BR/> throw(M)<BR/> end. <BR/> <BR/>enqueue(0, _) -><BR/> [];<BR/>enqueue(Num, Fd) -><BR/> case io:get_line(Fd, "") of<BR/> eof -><BR/> [];<BR/> Line -><BR/> Url = string:strip(Line, both, $\n),<BR/> {ok, RequestId} = http:request(head, {Url, []}, [{autoredirect, false}], [{sync, false}]),<BR/> Ids = enqueue(Num - 1, Fd),<BR/> [{RequestId, Url} | Ids]<BR/> end.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1160082843485994982006-10-05T23:14:00.000+02:002006-10-05T23:14:00.000+02:00Anonymous: There's concurrent programming and ther...Anonymous: There's concurrent programming and there's concurrent programming. Perl is good at "mildly concurrent, with real need for some library from CPAN" kind of concurrent programming. Erlang is good at "massively concurrent, with no need for fancy libraries" kind of concurrent programming (and Perl isn't even pretending to try here). The former is much more common than the latter - so Perl is a language of choice for a lot more concurrent programming tasks than Erlang.<BR/><BR/>Jason: Your program kills my machine. I guess it's simply reading urls file to memory all at once (Perl program does it line by line), then converting it to linked list of integers (memory usage x8), and then tokenizing it (memory usage x2).<BR/><BR/>Even if we converted the program to read lines one at a time, the x8 memory explosion for storing strings troubles me a lot. We cannot store result cache in memory any more, as it would massively swap (cache in Perl program is pretty big already, I just don't have 8x as much memory here). Does Erlang has some sort of packed string object ?tawhttps://www.blogger.com/profile/16972845140253292628noreply@blogger.comtag:blogger.com,1999:blog-27488238.post-1160081265639529932006-10-05T22:47:00.000+02:002006-10-05T22:47:00.000+02:00it means you should really call erlang with -run i...it means you should really call erlang with -run inets or inets:start() or application:start(inets) to start the inets application, you can use the client in sync or async mode<BR/><BR/>here is a quick example that handles 20 requests at a time like your example, but i'm sure more experienced erlers could make it much cleaner<BR/><BR/>-module(checker).<BR/>-compile(export_all).<BR/><BR/>start() -><BR/> Queue = read_urls(),<BR/> master_loop(Queue, []).<BR/><BR/>read_urls() -><BR/> {ok, Contents} = file:read_file("urls.txt"),<BR/> string:tokens(binary_to_list(Contents), "\n").<BR/><BR/>master_loop([],[]) -><BR/> done;<BR/>master_loop(Queue, []) -><BR/> {NewQ, Pending} = enqueue(20, Queue),<BR/> master_loop(NewQ, Pending);<BR/>master_loop(Queue, Pending) -><BR/> receive<BR/> {http, {RequestId, Result}} -><BR/> {value, {_R, Url}} = lists:keysearch(RequestId, 1, Pending),<BR/> {{_, Status, _}, _, _} = Result,<BR/> io:format("~s ~b~n", [Url, Status]),<BR/> {NewQ, NewIds} = enqueue(1, Queue),<BR/> Merged = [ {Id, U} || {Id, U} <- Pending, Id /= RequestId ] ++ NewIds,<BR/> master_loop(NewQ, Merged);<BR/> M -><BR/> throw(M)<BR/> end.<BR/><BR/>enqueue(_Num, []) -><BR/> {[],[]};<BR/>enqueue(0, Q) -><BR/> {Q, []};<BR/>enqueue(Num, [H|T]) -><BR/> {ok, RequestId} = http:request(head, {H, []}, [{autoredirect, false}], [{sync, false}]),<BR/> {NewQ, Ids} = enqueue(Num - 1, T),<BR/> {NewQ, [{RequestId, H}] ++ Ids}.Anonymousnoreply@blogger.com