SuckMT v.0.57

Version 0.57 was released on 2004-10-01

(C) 1999-2003 by Niels Basjes

Frequently Asked Questions
Download SuckMT
Whats new in version 0.57
The TODO list for future versions.

What ?

SuckMT is intended to be a multithreaded replacement for suck.
Note that SuckMT was inpired by suck but SuckMT has absolutely NO CODE IN COMMON with suck. !!

Why ?

The main problem I personally have with the original suck (currently maintained by Robert A. Yetman) is that it is single threaded.

At first glance this sounds like a silly reason to start a new implementation from scratch but I think it isn't. At home I have a 56K (== 5KB/s real download speed) telephone connection with a charge for each connected second (as do most people in the Netherlands). The problem I ran into is that when downloading news messages the original suck doesn't go above about 2KB/s. This means that I needed roughly 2.5 times more online time than was stricktly required.

When asking in nl.comp.os.linux on how to solve this problem people suggested to basically create two separate installations of suck and to manually spread the newsgroups between the two. This meant to me that something like this should be included into suck. After having had a quick look at the suck source code I descided that doing it from scratch would be better for my health. ;-)

How ?

The basic idea of SuckMT is that it has several Command handlers each with its own NNTP connection (all to the same server). All commandhandlers share the same threadsafe queue which contains a "todo-list". When a commandhandler is idle it gets the next command from the queue and starts to execute it. This way SuckMT automatically balances the download of all NEWS items over the available NNTP socket connections thus actually using all the available bandwidth of my 5KB/s modem line. When properly configured the actual maximum lies at about 8KB/s because a modem can do some pretty good compression on plain ASCII information like usenet messages.

Where ?

You can download SuckMT right here! and also some sample scripts.
SuckMT is released under the GNU Public License.

If you're using SuckMT please send me an email with your comments, likes and dislikes.

Don't forget to check the FAQ

Download SuckMT

Version	Release Date	Source	Precompiled Binaries
		All platforms	Linux-i386		Windows NT 4-i386
		All platforms	Dynamically linked	Statically linked	Statically linked
0.57	2004-10-01	.tar.gz .src.rpm	.rpm
0.56	2004-07-28	.tar.gz .src.rpm	.rpm
0.55	2003-04-30	.tar.gz .src.rpm	.rpm .tar.gz	.tar.gz
0.54	2001-08-28	.tar.gz .src.rpm	.rpm .tar.gz	.tar.gz	.zip
0.53	2000-10-23	.tar.gz .src.rpm	.rpm	.tar.gz	.zip
before 0.53	1999-2000	All files including the old ones

Sample Scripts

Because some people had problems posting to their local INN servers I decided to post my own config files and scripts related to this issue on this website. These files are my actual config files and they really work for me but they may not work for you mainly because you probably have a lot of things in different places. I have an older version of suck installed (for rpost) and these scripts assume it to be installed. These scripts have now been included in the 0.55 distribution but can also be downloaded here as SuckMT-sample-scripts.tar.gz.

History

There is also a TODO list of items I want to put in future versions of SuckMT.

Changes from 0.56 to 0.57
- Fixed: Serious bug in the INI file read code. Thanks to Bob Jaques for pointing it out to me. This bug effectively made suckmt unusable on many platforms. The strange thing is that the effected code has been in there since almost the very first version. I should have spotted this when making 0.56 but back then I only focussed on the makefiles and related scripts.

Changes from 0.55 to 0.56
In general this version has not changed the functionality at all. This is just a minor fixed version that allows for succesfull compilation on newer platforms (like Fedora Core 2).
- Fixed: Two issues regarding changed behaviour of automake:
- Added AC_PROG_CC to configure.in_ even though the C compiler is not used. Compilation would fail without it.
- Changed CFLAGS and CXXFLAGS into AM_CFLAGS and AM_CXXFLAGS dded initial support for BSD and Cygwin
- Fixed: Minor issue in the spec file (for building RPMS).

Changes from 0.54 to 0.55
In general this version has not changed the functionality significantly. Most changes are related to makefiles, RPM building, changed behaviour of the GCC compiler and packaging.
- Fixed: Some minor compiler warnings
- Updated: Added initial support for BSD and Cygwin
- Updated: Handling of the 423 error state (no such article number)
- Fixed: Some compiler problems with the FreeBSD port (Thanks to Charles F. Randall).
- Updated: Changed makefiles so they automatically force the usage of GNU make
- Fixed: Some errors and warnings that came to light with gcc3 (Yes, these were bugs in the code).
- Fixed: Some minor issues regarding GCC 3.2 as provided with Redhat 8.0 and 9.0
- Improved: It is now possible for any regular user to do make rpm.
- Improved: Included some of the improvements as provided by PLD.
- Added: Sample scripts are now part of the standard distribution

Changes from 0.53 to 0.54
- Added: Compiler flags for IRIX64 and Solaris (not fully tested).
- Added: Option to remove the ^M from the lines in the downloaded messages.
- Changed: Completely different regular expression library. Now uses the regex++ from www.boost.org which was created by Dr John Maddock. It is currently only enabled on Win32 due to an internal compiler error on my system (SuSE 6.1).
- Fixed: Usage of namespaces and C++ include files. I found that I really did it wrong in the previous versions.compiler:

Changes from 0.52 to 0.53
- Fixed: Uninitialized Memory Read in regular expression library.
- Changed: Can now use a "no posting" server (Thanks to Stephane Lentz)
- Updated: Some unimportant code layout stuff.
- Added: Compiler flags for IRIX (not fully tested).
- Added: The ini file can now contain comments that will be written back again.
- Fixed: forgotten #include <unistd.h> in DuplicatesChecker.cpp which didn't show up in SuSE 6.1
- Added: Maximum messages per group download feature if over limit.
- Changed: Default config file (if nothing is specified) is now /etc/suckmt.ini
- Changed: Default value for "Send MODE READER to the server" is now True.
- Improved: Installation prcedure now automagically includes a working config file.
- Fixed ?: Removed alarm call from GenericSocket.cpp, perhaps this solves the SIG_ALRM problem.
- Fixed ?: Removed regfree call from HeaderMatcher.cpp, perhaps this solves some unexplained crashes. It does introduce memory leaks at exit.
- Fixed: The mutexes in MultiStream.cpp which still allowed some race conditions.

Changes from 0.51 to 0.52
- Added: Retry option when the server says "Connection refused". The default value doesn't change 0.51 behaviour.
- Fixed: In case some of the connections fail strange things happened. This exotic situation is handled better now.
- Added: Better handling of "430 No such article"
- Fixed: Some compiler errors in Mandrake 6.0 (querying www.deja.com helps)
- Changed: Suppress the warnings from compiling the regex.c file (It's clear I didn't write that monstrosity).
- Fixed: Some of the statistics didn't add up correctly.
- Fixed: The messages on the screen from the different threads will no longer mix through each other even if one thread uses \r to stay on the same line.
- Fixed: Invalid message of 1 restart ID when none were present (Win32).
- Fixed: Didn't do a closesocket at the end.
- Changed: The NNTPGetArticleCommand can now be restarted when the connection failed.
- Changed: The way the connections got their number for the messages.

Changes from 0.50 to 0.51
- Fixed: The restart file didn't include the killed messages.
- Fixed: Constant size memory leak (unimportant but I fixed it anyway).
- Changed: The makefiles have been improved and it's now possible to enable debugging without having to change the makefiles.

Changes from 0.41 to 0.50
- You can check the progress when you do ps from a shell.
- Fixed Bug reading a value in the ini file that contained an = in the value.
- Added Restart file so it is safe to make the temp dir empty after an aborted download session.
- Added the headername "ArticleBody" can be used to create kill/keep rules that check for the actual article content.
- Added MultiStreams, all messages pass through a single stream type that can be configured to pass the information to the cerr, cout , the current logfile and the syslogd.
- Rewrite: (Almost) All messages use the LogStreams now.
- Changed: The -init option now updates the INI file if it already exists.
- Fixed: It is now possible to create an RPM and SRPM file.

Changes from 0.40 to 0.41
- SERIOUS BUG in 0.40 !!!! All statistics will be reset to 0 and all global kill rules will be set to -1 due to a serious bug in the read routine.

Changes from 0.3j to 0.40
- Regular expression matching of kill/keep rules.
- Case insensitive matching of kill/keep rules.
- Simple scoring system for all kill/keep rules.
- Added "Minimum Lines in a Message" as requested by Petri Laihonen.

Changes from 0.3i to 0.3j
- Changed the command syntax from just \n to \r\n and made this a centralized issue. I hope this fixes some problems that were reported regarding the authinfo.
- Added a simple debug option to track the socket commands.
- Created somewhat better makefiles.

Changes from 0.3h to 0.3i
- Created transparant boolean readout for IniFile.
- Made the NNTProxy query its parameters from an IniFile.
- Added the option for MODE READER
- Added the option for username and password to login using AUTHINFO
- Fixed lockup problem when aborting.

Changes from 0.3g to 0.3h
- Counting the messages missed some messages when errors ocurred

Changes from 0.3f to 0.3g
- Fixed a serious memory leak in NNTPGetArticleCommand when aborting.
- Minor fixes and resolution of compiler warnings (on SuSE 6.1: no warnings at all).
- Fixed some compilation problems that occur on Slackware (thanks to Dennis Bijwaard).

Changes from 0.3d to 0.3f
- Keep headers to overrule the kill headers.
- Kill logfile option.
- Re-port to NT to do some additional testing and debugging.

Changes from 0.3c to 0.3d
- Fixed a bug regarding the statistics number in the suckmt.ini file.
- Optimized the code a little (some kill checks moved to before downloading the header).

Version 0.3c
Does:
- work for me.
- Run on Linux 2.2.5 (SuSE 6.1)
- Download several newsgroups.
- Support transparent restarts.
- Generate output that can be post-processed by the same software as the original suck (I use innxmit).
- Supports killing on all kinds of headers and things like the maximum number of lines in the message and the maximum number of groups a message is crossposted to.
- Is portable to Windows NT (I did this only for testing and ensuring code portability).
- Include a installation guide.

Doesn't:
- Include a manual.
- Include a README file.
- Have tidy makefiles.
- do lots of other things which I don't realize yet....

Build and install ?

Just unpack the archive and run make in the source directory.
You'll get a single executable and that is everything.
Run the executable and you should be able to figure out the rest yourself.
The INI file that is generated can be editted using any plain text editor, if you see the file you'll know what the format is.
Read the INSTALL file for more detailled information.

Frequently Asked Questions

This is the FAQ for SuckMT. If you have a question not handled in the FAQ just mail me using SuckMT@Basjes.nl.

Q: I get lots of linking errors (unresolved externals) because only main.cpp is compiled and linked into the executable.
A: This is common to distributions like RedHat 5.1. Solution: You need to upgrade your automake package to version 1.4 or newer.

Q: SuckMT crashes when I try to download very large messages from a discussion group (multi megabyte ISO images).
A: SuckMT is designed to download the type of messages usenet was designed for: small discussion messages. Therefor SuckMT first downloads the entire message to memory an then writes it to disk. Downloading a huge message (like an ISO image in a single message) will most likely cause SuckMT to run into an "out of memory" situation. A different design for this is under consideration.

Q: Is there a precompiled static binary for SuckMT ?
A: Yes: Because I ran out of disk space at my provider there is only a precompiled static binary for the latest version (check the download section).

Q: Is there an RPM or SRPM for SuckMT ?
A: Yes: RPM and SRPM files for version 0.50 and newer are available in the download section.

Q: Is there a DEB (Debian Package) for SuckMT ?
A: No: I don't have a system with Debian on it and I don't have the time to install one just because it is a great distribution. If you want SuckMT on Debian either install the RPM (I've been told that can be done with a tool called alien) or compile the sources yourself.

Q: When I compile SuckMT on my SuSE system I get messages like this:
TraceLog.o: warning: multiple common of `MultiStream type_info node'
A: You can safely ignore these messages. The only real answer I've been able to find is located on http://clanlib.org/documentation/FAQ.html:

       To get rid of those annoying "multiple common..." warnings edit /usr/lib/gcc-lib/i486-linux/egcs-???/specs.

            *lib
            -warn-common%{share....

       should be:

            *lib
            %{share...

Q: Why aren't the files solaris.cc, mach.cc and nt.cpp compiled ?
A: SuckMT uses the GPL portable C++ multithreading library from the omniORB CORBA implementation. The 3 files are the platform specific multithreading implementations of this library. The distribution also contains Microsoft Visual C++ 6.0 project files to build a fully functional Windows executable that uses the nt.cpp instead of the posix.cc.

Q: Does SuckMT work on other platforms than Linux ?
A: SuckMT can be compiled out of the box for Linux and Windows NT. It should be a breeze to create makefiles for other UNIX systems but I haven't had the time or the machines to do that on.

Q: SuckMT crashes on my Slackware 4.0 system (libc 5). Is SuckMT really that buggy ?
A: No, it's not THAT buggy. I've had a few reports that SuckMT crashes on those systems. My best guess is that some part of either the socket or the streams library (or something like that) is not thread safe. The best remedy is to upgrade your system to a newer version runtime library because I've put great effort in making it as robust as possible.

Q: Can you add features to extract binaries (or something else) from the messages ?
A: No. SuckMT will ONLY transport the messages between your system and a newsserver. You should use an external tool if you want to extract specific information from the messages that have been downloaded.

Planned enhancements for future versions of SuckMT.

This is a list of items I plan to put in furture versions of SuckMT.
There are three sections here: "New features", "Improvements in the robustness/quality" and "Research into possible new features/ideas".

Before you get you hopes up of seeing any of these features in the next few weeks: I'm currently juggeling a full time (40 hour/week) job, a parttime (25 hour/week) university study and a new appartment that is in desparate need of decorating. Guess what... SuckMT will have to wait a little while (for now).
I you feel like adding to or changing something to SuckMT don't hesitate to fiddle around with the code. If you have a good idea for a new feature that is not listed below don't hesitate to contact me.

Planned Features

Multiple Servers: Make it possible to query serveral servers simultaneously for new messages. This will be handy for people who use several different (possibly unreliable) news servers to download the messages. This option will require a redesign of the internals of SuckMT and the suckmt.ini file because some groups may not be available on all servers and some will be available on several servers.
Uploading: Make it possible to upload the locally posted messages back to the news server of the provider. In combination with the multiple server thingy this should pose a challenge to think of a good way to choose where to post the messages; possibly post each message to all servers that carry the specified groups.
Hierarchical suckmt.ini file: I think that making it possible for the suckmt.ini file to be hierarchical should improve the ease at which I can include the Multiple servers and Uploading capabilities. NOTE: It's scary to see this part grow into .... the Windows registry ..... AAARRRGHHHH. ;-) .... hmmm, using this object oriented interface this would be a very nice way of making the storage of applications settings across platforms very portable and abstract.
History database: Let SuckMT query the history database of the local server (if available) to see if a message has been downloaded in a previous session. Perhaps I should .... re-use some relevant code from suck.
Sample scripts: I should really add some good sample scripts to the standard distribution.
GUI Tool: Having a nice (WEB based ?) GUI tool to edit the suckmt.ini file should be created to make it easier to add kill/keep rules, groups, servers, etc. from the platform where the user actually reads the news (in my case WinNT).
Don't shout: Do you ever get annoyed by people sending messages with a subject like "CAN SOMEONE PLEEZE HELP ME CUZ I A CERTIFIED LUZER WHO HAS JUST FOUND THE CAPSLOCK BUTTON" ? Well I think a kill feature whould be created for messages like this. The trouble is that sometimes an all caps subject is justified because it contains serveral acronyms or brandnames like SCSI and ASUS which should be written all caps. Perhaps this can be done with a nice regular expression ?
Limited number of messages: IMPLEMENTED IN v0.53 Limit the number of messages that is downloaded to a certain number. This is handy for new groups or if you just returned from holiday.

Planned improvements regarding the robustness and quality of SuckMT

Problems using rpost: IMPLEMENTED IN v0.54 The downloaded messages are stored exactly as received. This means that the end of each line has a "CR-LF" instead of just a "LF" (see section 2.4.1 of RFC 977). The fix should be that the downloaded messages have the "CR" removed before they are stored to disk.
Replace regexp: PARTIALLY IMPLEMENTED IN v0.54 Most unexplained crashes have been caused by the buggy regexp version I decided to use (at the time it was the best option available). The best candidate replacement is the one from http://www.boost.org. This one is real C++ and initial tests using Purify on Windows NT were very promising.
Occasional crash: Sometimes SuckMT crashes on my machine. I HATE THAT !!. I've had reports that SuckMT sometimes receives a signal 14 (SIGALRM: Alarm clock in POSIX) and sometimes an 'out of memory' (which may very well be the exact same problem). I haven't had the time to find the actual cause of this signal yet but they always seem to occur near the end of a rather large download session (more than 1500 messages). If someone knows why SuckMT crashes/receives this signal or (better yet) has a bug fix please let me know.
Compiler Warnings in SuSE: Each new version of SuSE seems to come up with a few more compiler warnings in SuckMT. I must find the real problem.
File/directory/security tests: Check the files and directories specified in the suckmt.ini file to see if they are really what they should be and if they can be written (security related).
Comments in the suckmt.ini file: IMPLEMENTED IN v0.53 The suckmt.ini file can handle comments just fine. The problem is that the comments are not stored so when the file is rewritten at the end of a succesfull session these comments are not written back and are therfore lost. The comments should be stored as well.
Sudden drop in the article numbers on the server: Sometimes a provider decides to start numbering a group from 0 again. The idea behind this enhancement is that is the high number of a group suddenly drops to a value which is significantly below the current number at which SuckMT should start loading, this number is automagically reset to 0 so SuckMT will download all the messages for that group.
Huge messages: If you download HUGE messages SuckMT can crash due to an out of memory. I should examine if it is possible to solve this possible crash, or perhaps I should ignore this problem.
Out of memory: Add an exception based handler for a out of memory situations (set_new_handler) that makes it possible for SuckMT to terminate cleanly without crashing.

Research into new features

Streaming use of innxmit: I have to look into this streaming mode of innxmit. Perhaps this can be usefull when locally posting the new messages. This is something for the sample scripts I want to include.
Hoard multiprocessor memory allocator: I think this library should be evaluated to see how much it adds to the speed on single and multiple CPU systems.

Comments

If you like, hate, have good ideas or have something else to tell me about SuckMT don't hesitate to let me know.