Lets pretend we have a user model with a created_at attribute which is stored as a timestamp.

If we want to plot a simple graph showing signups per date this is the way to extract them:

[cc lang="ruby"]
User.count(:group => “DATE(created_at)”)
[/cc]

Thats all there is to it. From this you will get an OrderedHash containing the date and count of users for each date.

One small thing to remember if you are using PostgreSQL. You will need the DATE() function in any ORDER/SELECT statements because PostgreSQL will only select/order by fields thats in the GROUP BY statement and this is a pretty healthy behavior that many databases (no names here) don’t care about. Think about it. If you are grouping all users by date and use * as selector how would the database know which username or email address to display?

This is what happens if you try to group by created_at without DATE() ->

[cc lang="ruby"]
>> User.count(:order => ‘created_at DESC’, :group => ["DATE(created_at)"])
ActiveRecord::StatementInvalid: PGError: ERROR: column “users.created_at” must appear in the GROUP BY clause or be used in an aggregate function
: SELECT count(*) AS count_all, DATE(created_at) AS date_created_at FROM “users” GROUP BY DATE(created_at) ORDER BY created_at DESC
from /tmp/app/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:212:in `log’
from /tmp/app/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:507:in `execute’
from /tmp/app/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:985:in `select_raw’
from /tmp/app/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:972:in `select’
from /tmp/app/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache’
from /tmp/app/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:62:in `select_all’
from /tmp/app/vendor/rails/activerecord/lib/active_record/calculations.rb:255:in `execute_grouped_calculation’
from /tmp/app/vendor/rails/activerecord/lib/active_record/calculations.rb:132:in `calculate’
from /tmp/app/vendor/rails/activerecord/lib/active_record/calculations.rb:130:in `catch’
from /tmp/app/vendor/rails/activerecord/lib/active_record/calculations.rb:130:in `calculate’
from /tmp/app/vendor/rails/activerecord/lib/active_record/calculations.rb:48:in `count’
from (irb):18
[/cc]

And this is the right way to do it:

[cc lang="ruby"]
>> User.count(:order => ‘DATE(created_at) DESC’, :group => ["DATE(created_at)"])
=> #22, “2009-10-25″=>4, “2009-11-04″=>8, “2009-10-15″=>9, “2009-10-26″=>16, “2009-11-05″=>9, “2009-10-16″=>193, “2009-10-27″=>14, “2009-11-06″=>9, “2009-10-17″=>49, “2009-10-28″=>15, “2009-11-07″=>6, “2009-10-18″=>36, “2009-10-29″=>8, “2009-10-19″=>116, “2009-10-30″=>15>[/cc]

As you may notice the Hash does not look ordered. But if you loop over it you will get them in order:

[cc lang='ruby']
>> User.count(:order => ‘DATE(created_at) DESC’, :group => ["DATE(created_at)"]).each {|u| puts “#{u[0]} -> #{u[1]}” }
2009-11-07 -> 6
2009-11-06 -> 9
2009-11-05 -> 9
2009-11-04 -> 8
2009-11-03 -> 14
2009-11-02 -> 20
2009-11-01 -> 10
2009-10-31 -> 6
2009-10-30 -> 15
2009-10-29 -> 8
..
[/cc]

And thats how you group a timestamp field with just its date part.

Cheers!

This is my way of solving it.

[cc console]
$ couchdb
kernel-poll not supported; “K” parameter ignored
Apache CouchDB 0.9.0 (LogLevel=info) is starting.
{“init terminating in do_boot”,{{badmatch,{error,shutdown}},[{couch_server_sup,start_server,1},{erl_eval,do_apply,5},{erl_eval,exprs,5},{init,start_it,1},{init,start_em,1}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()
[/cc]

Even with debug turned on it’s pretty hard to understand.

[cc console]
kernel-poll not supported; “K” parameter ignored
Apache CouchDB 0.9.0 (LogLevel=debug) is starting.
Configuration Settings ["./couchdb/local.ini"]:
[Couch] BindAddress=”127.0.0.1″
[Couch] ConsoleStartupMsg=”Apache CouchDB is starting.”
[Couch] DbRootDir=”/home/mathias/couchdb”
[Couch] DocumentRoot=”/home/mathias/couchdb/www”
[Couch] LogFile=”/home/mathias/couchdb/couch.log”
[Couch] LogLevel=”debug”
[Couch] Port=”5984″
[Couch] UtilDriverDir=”/usr/local/lib/couchdb/erlang/lib/couch-0.8.1-incubating/priv/lib”
[Couch Query Servers] javascript=”/usr/local/bin/couchjs /usr/local/share/couchdb/server/main.js”
[couchdb] database_dir=”/home/mathias/couchdb”
[couchdb] view_index_dir=”/home/mathias/couchdb”
[httpd] bind_address=”127.0.0.1″
[httpd] port=”5984″
[log] level=”debug”
[error] [<0.41.0>] {error_report,<0.21.0>,
{<0.41.0>,crash_report,
[[{pid,<0.41.0>},
{registered_name,couch_server},
{error_info,
{exit,
{badarg,
[{erlang,list_to_integer,[undefined]},
{couch_server,init,1},
{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]},
[{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},
{initial_call,{couch_server,init,['Argument__1']}},
{ancestors,[couch_primary_services,couch_server_sup,<0.1.0>]},
{messages,[]},
{links,[<0.36.0>]},
{dictionary,[]},
{trap_exit,false},
{status,running},
{heap_size,233},
{stack_size,23},
{reductions,113}],
[]]}}
{“init terminating in do_boot”,{{badmatch,{error,shutdown}},[{couch_server_sup,start_server,1},{erl_eval,do_apply,5},{erl_eval,exprs,5},{init,start_it,1},{init,start_em,1}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()
[/cc]

When running things as unprivileged users the usual problems are that you do not have access to read/write files or binding to the lower ports A.K.A. “well known ports”. With that in mind I located the original config files for couchdb with the plan of copying or overriding them.

My config files where located under /usr/local/etc/couchdb/

[cc console]
$ ls -lsa /usr/local/etc/couchdb/
4 -rw-r–r– 1 root root 2626 2009-04-10 11:58 default.ini
4 -rw-r–r– 1 root root 847 2009-04-10 14:17 local.ini
[/cc]

Reading the documentation I found out that local.ini is just for overriding stuff and local.ini will never be overwritten by any upgrades. The config files contain lots of settings, but my aim is just to get it to start so I focus on the things that I think is the problem and thats all the PATH settings.

In my unprivileged users home folder I created a folder named couchdb to hold my config and datafiles. I also created a www folder inside it.

After that I create my local.ini to override my settings:

[cc console]
$ cat couchdb/local.ini

[couchdb]
database_dir = ./couchdb
view_index_dir = ./couchdb

[httpd]
port = 5984
bind_address = 127.0.0.1

[log]
level = info
file=./couchdb/couch.log
[/cc]

When our new config is in place we can try to start couchdb again.

[cc console]
$ couchdb -C ./couchdb/local.ini -p ./couchdb/couch.pid -b
Apache CouchDB has started, time to relax.
[/cc]

-C means use configuration FILE (chainable, does not reset system default). If we do not want to use the system config at all we can use a lowercase -c but we want to keep our overridden config as small as possible right now.

-p is the path to our pid file and -b tells couchdb to spawn in the background.

If you are on your local machine you can just launch your browser and point it to http://127.0.0.1:5984/

[cc json]
{“couchdb”:”Welcome”,”version”:”0.9.0″}
[/cc]

or you can access the admin gui trough: http://127.0.0.1:5984/_utils/index.html.

After I started to write this blogpost I did found a page on the Couchdb wiki that explains Error messages and solutions and here it is: http://wiki.apache.org/couchdb/Error_messages.

Happy haxxing!

pgFouine is pretty simple to setup and the process is where well documented at their website (http://pgfouine.projects.postgresql.org/tutorial.html).

However syslog-ng is very different if you are used to “normal” syslog daemon.

The first thing you have to do is setup PostgreSQL as the instructions says on the website. The next step is to tell syslog-ng to log you postgres data to a separate file.

The first thing to create in syslog-ng.conf is a destination. Destination blocks is used to send logs somewhere.

By adding the following row we tell syslog-ng that there is a destination called postgres and that we want this data in /var/log/pgsql.

[cc lang="bash"]
destination postgres { file(“/var/log/pgsql”); };
[/cc]

Now we are ready to add a filter to caputre our PostgreSQL data. Filters can both be used to include or exclude data from you logs. The following row in your syslog-conf captures all data sent to local0 or whatever you called it in postgresql.conf

[cc lang="bash"]
filter f_postgres { facility(local0); };
[/cc]

It’s time to join our destination with our filter and the following line tells syslog-ng to join our source, destination and filter together.

[cc lang="bash]
log { source(src); filter(f_postgres); destination(postgres); };
[/cc]

Put these lines in your syslog-ng.conf and restart syslog-ng and also PostgreSQL and you will have logging in a separate file!

Next step is to write a little guide to pgFouine

Cheers!

It happens that we are asked to do some work on our customers dedicated servers and when that happens it’s not that uncommon that there is not a single person alive that has the root password for the MySQL server. When that happen it’s good to know how to reset it

This post is about how simple it’s to reset MySQL’s root password under FreeBSD.

The first step is to start MySQL without it’s grant tables.

Under FreeBSD you pass your mysql startup parameters in /etc/rc.conf

Add the following to /etc/rc.conf. We add –skip-networking to because we do not want someone from the outside to connect without passwords to our server.

[cc lang="bash"]
mysql_enable=”YES”
mysql_args=”–skip-grant-tables –skip-networking”
[/cc]

And then restart MySQL with the following command.

[cc lang="bash"]
> /usr/local/etc/rc.d/mysql-server restart
[/cc]

When MySQL comes back online it’s running without grant tables which means that you do not have to use a password to connect to it (no one has to).

Simply connect as root (without password) and issue the following SQL query.

[cc lang="sql"]
> mysql -u root mysql

mysql> UPDATE mysql.user SET Password=PASSWORD(‘‘) WHERE User=’root’;
[/cc]

Now we are almost done. Just comment the skip-grant-tables line from rc.conf and restart MySQL and you should have a brand new root password.

For for other *nix users the process is the same except from how you pass your MySQL params to MySQL at startup. You can also start MySQL manually with something like this.

[cc lang="bash"]

• SSD vs. SATA RAID: A performance benchmark
• SSD vs. SATA benchmarks, round 2: Server applications

From within the database of choice.

Run the following query to get some useful table statistics.
[cc lang="html"]
SELECT relname, reltuples, relpages * 8 / 1024 AS “MB” FROM pg_class ORDER BY relpages DESC;
[/cc]

Block size

If you use another block size than 8kb you have do multiply relpages with your own block size. To find your block size you can use pg_controldata which is a command-line utility to display control information of a PostgreSQL database cluster.

Run it with your datadir as first parameter pg_controldata ./data/ and look for Database block size in the output.

Database query result

The outout gives you:

1. relname: The name of the table.
2. reltuples: Estimated numbers of rows.
3. MB: Table size in Mbytes.

[cc lang="html"]
relname | reltuples | MB
———————————–+————-+——-
history_uint | 1.685e+08 | 15396
history_uint_1 | 1.685e+08 | 12279
history | 1.40719e+08 | 8405
history_1 | 1.40719e+08 | 7153
trends | 1.03077e+07 | 929
trends_pkey | 1.03077e+07 | 423
items | 4698 | 83
history_str | 536599 | 53
history_str_1 | 536599 | 27
events | 166602 | 15
[/cc]

We can now see that history_uint is the biggest table and holds an estimate of 168,572,082 rows and takes about 15GB on disk.

Be aware that these numbers only gets updated when you run vacuum.

I have more PostgreSQL internals to come!