PentHertz
/
CommonLibs
mirror of https://github.com/PentHertz/CommonLibs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

syncing CommonLibs, and including BSD-licensed A5/1 implementation. 

git-svn-id: http://wush.net/svn/range/software/public/CommonLibs/trunk@6165 19bc5d8c-e614-43d4-8b26-e1612bc8e597
This commit is contained in:
Kurtis Heimerl 2013-08-13 23:17:46 +00:00
parent 59515c11f0
commit 7a7862c9d5
15 changed files with 684 additions and 90 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

228
A51.cpp Normal file
View File

@ -0,0 +1,228 @@
/*
* A pedagogical implementation of A5/1.
*
* Copyright (C) 1998-1999: Marc Briceno, Ian Goldberg, and David Wagner
*
* The source code below is optimized for instructional value and clarity.
* Performance will be terrible, but that's not the point.
* The algorithm is written in the C programming language to avoid ambiguities
* inherent to the English language. Complain to the 9th Circuit of Appeals
* if you have a problem with that.
*
* This software may be export-controlled by US law.
*
* This software is free for commercial and non-commercial use as long as
* the following conditions are aheared to.
* Copyright remains the authors' and as such any Copyright notices in
* the code are not to be removed.
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* The license and distribution terms for any publicly available version or
* derivative of this code cannot be changed. i.e. this code cannot simply be
* copied and put under another distribution license
* [including the GNU Public License.]
*
* Background: The Global System for Mobile communications is the most widely
* deployed cellular telephony system in the world. GSM makes use of
* four core cryptographic algorithms, neither of which has been published by
* the GSM MOU. This failure to subject the algorithms to public review is all
* the more puzzling given that over 100 million GSM
* subscribers are expected to rely on the claimed security of the system.
*
* The four core GSM algorithms are:
* A3 authentication algorithm
* A5/1 "strong" over-the-air voice-privacy algorithm
* A5/2 "weak" over-the-air voice-privacy algorithm
* A8 voice-privacy key generation algorithm
*
* In April of 1998, our group showed that COMP128, the algorithm used by the
* overwhelming majority of GSM providers for both A3 and A8
* functionality was fatally flawed and allowed for cloning of GSM mobile
* phones.
* Furthermore, we demonstrated that all A8 implementations we could locate,
* including the few that did not use COMP128 for key generation, had been
* deliberately weakened by reducing the keyspace from 64 bits to 54 bits.
* The remaining 10 bits are simply set to zero!
*
* See <A HREF="http://www.scard.org/gsm">http://www.scard.org/gsm</A> for additional information.
*
* The question so far unanswered is if A5/1, the "stronger" of the two
* widely deployed voice-privacy algorithm is at least as strong as the
* key. Meaning: "Does A5/1 have a work factor of at least 54 bits"?
* Absent a publicly available A5/1 reference implementation, this question
* could not be answered. We hope that our reference implementation below,
* which has been verified against official A5/1 test vectors, will provide
* the cryptographic community with the base on which to construct the
* answer to this important question.
*
* Initial indications about the strength of A5/1 are not encouraging.
* A variant of A5, while not A5/1 itself, has been estimated to have a
* work factor of well below 54 bits. See http://jya.com/crack-a5.htm for
* background information and references.
*
* With COMP128 broken and A5/1 published below, we will now turn our attention
* to A5/2. The latter has been acknowledged by the GSM community to have
* been specifically designed by intelligence agencies for lack of security.
*
*/
#include <stdlib.h>
#include <assert.h>
#include "A51.h"
/* Masks for the three shift registers */
#define R1MASK 0x07FFFF /* 19 bits, numbered 0..18 */
#define R2MASK 0x3FFFFF /* 22 bits, numbered 0..21 */
#define R3MASK 0x7FFFFF /* 23 bits, numbered 0..22 */
/* Middle bit of each of the three shift registers, for clock control */
#define R1MID 0x000100 /* bit 8 */
#define R2MID 0x000400 /* bit 10 */
#define R3MID 0x000400 /* bit 10 */
/* The three shift registers. They're in global variables to make the code
* easier to understand.
* A better implementation would not use global variables. */
word R1, R2, R3;
/* Look at the middle bits of R1,R2,R3, take a vote, and
* return the majority value of those 3 bits. */
bit majority() {
int sum;
sum = ((R1>>8)&1) + ((R2>>10)&1) + ((R3>>10)&1);
if (sum >= 2)
return 1;
else
return 0;
}
/* Clock two or three of R1,R2,R3, with clock control
* according to their middle bits.
* Specifically, we clock Ri whenever Ri's middle bit
* agrees with the majority value of the three middle bits.*/
void clock() {
bit maj = majority();
if (((R1&R1MID)!=0) == maj)
R1 = ((R1<<1) & R1MASK) | (1 & (R1>>18 ^ R1>>17 ^ R1>>16 ^ R1>>13));
if (((R2&R2MID)!=0) == maj)
R2 = ((R2<<1) & R2MASK) | (1 & (R2>>21 ^ R2>>20));
if (((R3&R3MID)!=0) == maj)
R3 = ((R3<<1) & R3MASK) | (1 & (R3>>22 ^ R3>>21 ^ R3>>20 ^ R3>>7));
}
/* Clock all three of R1,R2,R3, ignoring their middle bits.
* This is only used for key setup. */
void clockallthree() {
R1 = ((R1<<1) & R1MASK) | (1 & (R1>>18 ^ R1>>17 ^ R1>>16 ^ R1>>13));
R2 = ((R2<<1) & R2MASK) | (1 & (R2>>21 ^ R2>>20));
R3 = ((R3<<1) & R3MASK) | (1 & (R3>>22 ^ R3>>21 ^ R3>>20 ^ R3>>7));
}
/* Generate an output bit from the current state.
* You grab a bit from each register via the output generation taps;
* then you XOR the resulting three bits. */
bit getbit() {
return ((R1>>18)^(R2>>21)^(R3>>22))&1;
}
/* Do the A5/1 key setup. This routine accepts a 64-bit key and
* a 22-bit frame number. */
void keysetup(byte key[8], word frame) {
int i;
bit keybit, framebit;
/* Zero out the shift registers. */
R1 = R2 = R3 = 0;
/* Load the key into the shift registers,
* LSB of first byte of key array first,
* clocking each register once for every
* key bit loaded. (The usual clock
* control rule is temporarily disabled.) */
for (i=0; i<64; i++) {
clockallthree(); /* always clock */
keybit = (key[i/8] >> (i&7)) & 1; /* The i-th bit of the
key */
R1 ^= keybit; R2 ^= keybit; R3 ^= keybit;
}
/* Load the frame number into the shift
* registers, LSB first,
* clocking each register once for every
* key bit loaded. (The usual clock
* control rule is still disabled.) */
for (i=0; i<22; i++) {
clockallthree(); /* always clock */
framebit = (frame >> i) & 1; /* The i-th bit of the frame #
*/
R1 ^= framebit; R2 ^= framebit; R3 ^= framebit;
}
/* Run the shift registers for 100 clocks
* to mix the keying material and frame number
* together with output generation disabled,
* so that there is sufficient avalanche.
* We re-enable the majority-based clock control
* rule from now on. */
for (i=0; i<100; i++) {
clock();
}
/* Now the key is properly set up. */
}
/* Generate output. We generate 228 bits of
* keystream output. The first 114 bits is for
* the A->B frame; the next 114 bits is for the
* B->A frame. You allocate a 15-byte buffer
* for each direction, and this function fills
* it in. */
void run(byte AtoBkeystream[], byte BtoAkeystream[]) {
int i;
/* Zero out the output buffers. */
for (i=0; i<=113/8; i++)
AtoBkeystream[i] = BtoAkeystream[i] = 0;
/* Generate 114 bits of keystream for the
* A->B direction. Store it, MSB first. */
for (i=0; i<114; i++) {
clock();
AtoBkeystream[i/8] |= getbit() << (7-(i&7));
}
/* Generate 114 bits of keystream for the
* B->A direction. Store it, MSB first. */
for (i=0; i<114; i++) {
clock();
BtoAkeystream[i/8] |= getbit() << (7-(i&7));
}
}
void A51_GSM( byte *key, int klen, int count, byte *block1, byte *block2 )
{
assert(klen == 64);
keysetup(key, count); // TODO - frame and count are not the same
run(block1, block2);
}

11
A51.h Normal file
View File

@ -0,0 +1,11 @@
#include <stdio.h>
#include <stdlib.h>
typedef unsigned char byte;
typedef unsigned long word;
typedef word bit;
void A51_GSM( byte *key, int klen, int count, byte *block1, byte *block2 );

162
A51Test.cpp Normal file
View File

@ -0,0 +1,162 @@
/*
* A pedagogical implementation of A5/1.
*
* Copyright (C) 1998-1999: Marc Briceno, Ian Goldberg, and David Wagner
*
* The source code below is optimized for instructional value and clarity.
* Performance will be terrible, but that's not the point.
* The algorithm is written in the C programming language to avoid ambiguities
* inherent to the English language. Complain to the 9th Circuit of Appeals
* if you have a problem with that.
*
* This software may be export-controlled by US law.
*
* This software is free for commercial and non-commercial use as long as
* the following conditions are aheared to.
* Copyright remains the authors' and as such any Copyright notices in
* the code are not to be removed.
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* The license and distribution terms for any publicly available version or
* derivative of this code cannot be changed. i.e. this code cannot simply be
* copied and put under another distribution license
* [including the GNU Public License.]
*
* Background: The Global System for Mobile communications is the most widely
* deployed cellular telephony system in the world. GSM makes use of
* four core cryptographic algorithms, neither of which has been published by
* the GSM MOU. This failure to subject the algorithms to public review is all
* the more puzzling given that over 100 million GSM
* subscribers are expected to rely on the claimed security of the system.
*
* The four core GSM algorithms are:
* A3 authentication algorithm
* A5/1 "strong" over-the-air voice-privacy algorithm
* A5/2 "weak" over-the-air voice-privacy algorithm
* A8 voice-privacy key generation algorithm
*
* In April of 1998, our group showed that COMP128, the algorithm used by the
* overwhelming majority of GSM providers for both A3 and A8
* functionality was fatally flawed and allowed for cloning of GSM mobile
* phones.
* Furthermore, we demonstrated that all A8 implementations we could locate,
* including the few that did not use COMP128 for key generation, had been
* deliberately weakened by reducing the keyspace from 64 bits to 54 bits.
* The remaining 10 bits are simply set to zero!
*
* See <A HREF="http://www.scard.org/gsm">http://www.scard.org/gsm</A> for additional information.
*
* The question so far unanswered is if A5/1, the "stronger" of the two
* widely deployed voice-privacy algorithm is at least as strong as the
* key. Meaning: "Does A5/1 have a work factor of at least 54 bits"?
* Absent a publicly available A5/1 reference implementation, this question
* could not be answered. We hope that our reference implementation below,
* which has been verified against official A5/1 test vectors, will provide
* the cryptographic community with the base on which to construct the
* answer to this important question.
*
* Initial indications about the strength of A5/1 are not encouraging.
* A variant of A5, while not A5/1 itself, has been estimated to have a
* work factor of well below 54 bits. See http://jya.com/crack-a5.htm for
* background information and references.
*
* With COMP128 broken and A5/1 published below, we will now turn our attention
* to A5/2. The latter has been acknowledged by the GSM community to have
* been specifically designed by intelligence agencies for lack of security.
*
*/
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include "./A51.h"
/* Test the code by comparing it against
* a known-good test vector. */
void test() {
byte key[8] = {0x12, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF};
word frame = 0x134;
byte goodAtoB[15] = { 0x53, 0x4E, 0xAA, 0x58, 0x2F, 0xE8, 0x15,
0x1A, 0xB6, 0xE1, 0x85, 0x5A, 0x72, 0x8C, 0x00 };
byte goodBtoA[15] = { 0x24, 0xFD, 0x35, 0xA3, 0x5D, 0x5F, 0xB6,
0x52, 0x6D, 0x32, 0xF9, 0x06, 0xDF, 0x1A, 0xC0 };
byte AtoB[15], BtoA[15];
int i, failed=0;
A51_GSM(key, 64, frame, AtoB, BtoA);
/* Compare against the test vector. */
for (i=0; i<15; i++)
if (AtoB[i] != goodAtoB[i])
failed = 1;
for (i=0; i<15; i++)
if (BtoA[i] != goodBtoA[i])
failed = 1;
/* Print some debugging output. */
printf("key: 0x");
for (i=0; i<8; i++)
printf("%02X", key[i]);
printf("\n");
printf("frame number: 0x%06X\n", (unsigned int)frame);
printf("known good output:\n");
printf(" A->B: 0x");
for (i=0; i<15; i++)
printf("%02X", goodAtoB[i]);
printf(" B->A: 0x");
for (i=0; i<15; i++)
printf("%02X", goodBtoA[i]);
printf("\n");
printf("observed output:\n");
printf(" A->B: 0x");
for (i=0; i<15; i++)
printf("%02X", AtoB[i]);
printf(" B->A: 0x");
for (i=0; i<15; i++)
printf("%02X", BtoA[i]);
printf("\n");
if (!failed) {
printf("Self-check succeeded: everything looks ok.\n");
} else {
/* Problems! The test vectors didn't compare*/
printf("\nI don't know why this broke; contact the authors.\n");
exit(1);
}
printf("time test\n");
int n = 10000;
float t = clock();
for (i = 0; i < n; i++) {
A51_GSM(key, 64, frame, AtoB, BtoA);
}
t = (clock() - t) / (CLOCKS_PER_SEC * (float)n);
printf("A51_GSM takes %g seconds per iteration\n", t);
}
int main(void) {
test();
return 0;
}

23
BitVector.h
View File

@ -351,9 +351,12 @@ class BitVector : public Vector<char> {
other.mData=NULL;
}
void settfb(int i, int j) const
/** Set a bit */
void settfb(size_t index, int value)
{
mStart[i] = j;
char *dp = mStart+index;
assert(dp<mEnd);
*dp = value;
}
};
@ -449,6 +452,22 @@ class SoftVector: public Vector<float> {
/** Slice the whole signal into bits. */
BitVector sliced() const;
/** Return a soft bit. */
float softbit(size_t index) const
{
const float *dp = mStart+index;
assert(dp<mEnd);
return *dp;
}
/** Set a soft bit */
void settfb(size_t index, float value)
{
float *dp = mStart+index;
assert(dp<mEnd);
*dp = value;
}
};

67
Configuration.cpp
View File

@ -32,12 +32,6 @@
#include <iostream>
#include <string.h>
#ifdef DEBUG_CONFIG
#define debugLogEarly gLogEarly
#else
#define debugLogEarly
#endif
using namespace std;
@ -84,9 +78,24 @@ ConfigurationTable::ConfigurationTable(const char* filename, const char *wCmdNam
if (!sqlite3_command(mDB,createConfigTable)) {
gLogEarly(LOG_EMERG, "cannot create configuration table in database at %s, error message: %s", filename, sqlite3_errmsg(mDB));
}
// Set high-concurrency WAL mode.
if (!sqlite3_command(mDB,enableWAL)) {
gLogEarly(LOG_EMERG, "cannot enable WAL mode on database at %s, error message: %s", filename, sqlite3_errmsg(mDB));
}
// Build CommonLibs schema
ConfigurationKey *tmp;
tmp = new ConfigurationKey("Control.NumSQLTries","3",
"attempts",
ConfigurationKey::DEVELOPER,
ConfigurationKey::VALRANGE,
"1:10",// educated guess
false,
"Number of times to retry SQL queries before declaring a database access failure."
);
mSchema[tmp->getName()] = *tmp;
delete tmp;
tmp = new ConfigurationKey("Log.Alarms.Max","20",
"alarms",
ConfigurationKey::CUSTOMER,
@ -96,7 +105,7 @@ ConfigurationTable::ConfigurationTable(const char* filename, const char *wCmdNam
"Maximum number of alarms to remember inside the application."
);
mSchema[tmp->getName()] = *tmp;
free(tmp);
delete tmp;
tmp = new ConfigurationKey("Log.File","",
"",
@ -110,7 +119,7 @@ ConfigurationTable::ConfigurationTable(const char* filename, const char *wCmdNam
"To disable again, execute \"unconfig Log.File\"."
);
mSchema[tmp->getName()] = *tmp;
free(tmp);
delete tmp;
tmp = new ConfigurationKey("Log.Level","NOTICE",
"",
@ -128,7 +137,7 @@ ConfigurationTable::ConfigurationTable(const char* filename, const char *wCmdNam
"Default logging level when no other level is defined for a file."
);
mSchema[tmp->getName()] = *tmp;
free(tmp);
delete tmp;
// Add application specific schema
mSchema.insert(wSchema.begin(), wSchema.end());
@ -150,12 +159,13 @@ string ConfigurationTable::getDefaultSQL(const std::string& program, const std::
ss << "-- rather in the program's ConfigurationKey schema." << endl;
ss << "--" << endl;
ss << "PRAGMA foreign_keys=OFF;" << endl;
ss << "PRAGMA journal_mode=WAL;" << endl;
ss << "BEGIN TRANSACTION;" << endl;
ss << "CREATE TABLE CONFIG ( KEYSTRING TEXT UNIQUE NOT NULL, VALUESTRING TEXT, STATIC INTEGER DEFAULT 0, OPTIONAL INTEGER DEFAULT 0, COMMENTS TEXT DEFAULT '');" << endl;
ss << "CREATE TABLE IF NOT EXISTS CONFIG ( KEYSTRING TEXT UNIQUE NOT NULL, VALUESTRING TEXT, STATIC INTEGER DEFAULT 0, OPTIONAL INTEGER DEFAULT 0, COMMENTS TEXT DEFAULT '');" << endl;
mp = mSchema.begin();
while (mp != mSchema.end()) {
ss << "INSERT INTO \"CONFIG\" VALUES(";
ss << "INSERT OR IGNORE INTO \"CONFIG\" VALUES(";
// name
ss << "'" << mp->first << "',";
// default
@ -269,7 +279,8 @@ bool ConfigurationTable::defines(const string& key)
ScopedLock lock(mLock);
return lookup(key).defined();
} catch (ConfigurationTableKeyNotFound) {
debugLogEarly(LOG_ALERT, "configuration parameter %s not found", key.c_str());
// TODO: re-enable once we figure out why this message is being sent to syslog regardless of log level
//gLogEarly(LOG_DEBUG, "configuration parameter %s not found", key.c_str());
return false;
}
}
@ -580,7 +591,7 @@ string ConfigurationTable::getStr(const string& key)
return lookup(key).value();
} catch (ConfigurationTableKeyNotFound) {
// Raise an alert and re-throw the exception.
debugLogEarly(LOG_ALERT, "configuration parameter %s has no defined value", key.c_str());
gLogEarly(LOG_DEBUG, "configuration parameter %s has no defined value", key.c_str());
throw ConfigurationTableKeyNotFound(key);
}
}
@ -592,7 +603,7 @@ bool ConfigurationTable::getBool(const string& key)
return getNum(key) != 0;
} catch (ConfigurationTableKeyNotFound) {
// Raise an alert and re-throw the exception.
debugLogEarly(LOG_ALERT, "configuration parameter %s has no defined value", key.c_str());
gLogEarly(LOG_DEBUG, "configuration parameter %s has no defined value", key.c_str());
throw ConfigurationTableKeyNotFound(key);
}
}
@ -606,7 +617,7 @@ long ConfigurationTable::getNum(const string& key)
return lookup(key).number();
} catch (ConfigurationTableKeyNotFound) {
// Raise an alert and re-throw the exception.
debugLogEarly(LOG_ALERT, "configuration parameter %s has no defined value", key.c_str());
gLogEarly(LOG_DEBUG, "configuration parameter %s has no defined value", key.c_str());
throw ConfigurationTableKeyNotFound(key);
}
}
@ -619,7 +630,7 @@ float ConfigurationTable::getFloat(const string& key)
return lookup(key).floatNumber();
} catch (ConfigurationTableKeyNotFound) {
// Raise an alert and re-throw the exception.
debugLogEarly(LOG_ALERT, "configuration parameter %s has no defined value", key.c_str());
gLogEarly(LOG_DEBUG, "configuration parameter %s has no defined value", key.c_str());
throw ConfigurationTableKeyNotFound(key);
}
}
@ -634,7 +645,7 @@ std::vector<string> ConfigurationTable::getVectorOfStrings(const string& key)
line = strdup(rec.value().c_str());
} catch (ConfigurationTableKeyNotFound) {
// Raise an alert and re-throw the exception.
debugLogEarly(LOG_ALERT, "configuration parameter %s has no defined value", key.c_str());
gLogEarly(LOG_DEBUG, "configuration parameter %s has no defined value", key.c_str());
throw ConfigurationTableKeyNotFound(key);
}
@ -665,7 +676,7 @@ std::vector<unsigned> ConfigurationTable::getVector(const string& key)
line = strdup(rec.value().c_str());
} catch (ConfigurationTableKeyNotFound) {
// Raise an alert and re-throw the exception.
debugLogEarly(LOG_ALERT, "configuration parameter %s has no defined value", key.c_str());
gLogEarly(LOG_DEBUG, "configuration parameter %s has no defined value", key.c_str());
throw ConfigurationTableKeyNotFound(key);
}
@ -753,7 +764,13 @@ bool ConfigurationTable::set(const string& key, const string& value)
{
assert(mDB);
ScopedLock lock(mLock);
string cmd = "INSERT OR REPLACE INTO CONFIG (KEYSTRING,VALUESTRING,OPTIONAL) VALUES (\"" + key + "\",\"" + value + "\",1)";
string cmd;
if (keyDefinedInSchema(key)) {
cmd = "INSERT OR REPLACE INTO CONFIG (KEYSTRING,VALUESTRING,OPTIONAL,COMMENTS) VALUES (\"" + key + "\",\"" + value + "\",1,\'" + mSchema[key].getDescription() + "\')";
} else {
cmd = "INSERT OR REPLACE INTO CONFIG (KEYSTRING,VALUESTRING,OPTIONAL) VALUES (\"" + key + "\",\"" + value + "\",1)";
}
bool success = sqlite3_command(mDB,cmd.c_str());
// Cache the result.
if (success) mCache[key] = ConfigurationRecord(value);
@ -767,18 +784,6 @@ bool ConfigurationTable::set(const string& key, long value)
return set(key,buffer);
}
bool ConfigurationTable::set(const string& key)
{
assert(mDB);
ScopedLock lock(mLock);
string cmd = "INSERT OR REPLACE INTO CONFIG (KEYSTRING,VALUESTRING,OPTIONAL) VALUES (\"" + key + "\",NULL,1)";
bool success = sqlite3_command(mDB,cmd.c_str());
if (success) mCache[key] = ConfigurationRecord(true);
return success;
}
void ConfigurationTable::checkCacheAge()
{
// mLock is set by caller

4
Configuration.h
View File

@ -173,6 +173,7 @@ typedef std::map<std::string, ConfigurationRecord> ConfigurationRecordMap;
typedef std::map<HashString, ConfigurationRecord> ConfigurationMap;
class ConfigurationKey;
typedef std::map<std::string, ConfigurationKey> ConfigurationKeyMap;
ConfigurationKeyMap getConfigurationKeys();
/**
A class for maintaining a configuration key-value table,
@ -263,9 +264,6 @@ class ConfigurationTable {
/** Set or change a value in the table. */
bool set(const std::string& key, long value);
/** Create an entry in the table, no value though. */
bool set(const std::string& key);
/**
Remove an entry from the table.
Will not alter required values.

6
ConfigurationTest.cpp
View File

@ -121,7 +121,7 @@ ConfigurationKeyMap getConfigurationKeys()
""
);
map[tmp->getName()] = *tmp;
free(tmp);
delete tmp;
tmp = new ConfigurationKey("numnumber","42",
"",
@ -132,7 +132,7 @@ ConfigurationKeyMap getConfigurationKeys()
""
);
map[tmp->getName()] = *tmp;
free(tmp);
delete tmp;
tmp = new ConfigurationKey("newstring","new string value",
"",
@ -143,7 +143,7 @@ ConfigurationKeyMap getConfigurationKeys()
""
);
map[tmp->getName()] = *tmp;
free(tmp);
delete tmp;
return map;
}

12
Makefile.am
View File

@ -41,7 +41,8 @@ libcommon_la_SOURCES = \
Configuration.cpp \
sqlite3util.cpp \
URLEncode.cpp \
Utils.cpp
Utils.cpp \
A51.cpp
noinst_PROGRAMS = \
BitVectorTest \
@ -53,7 +54,8 @@ noinst_PROGRAMS = \
ConfigurationTest \
LogTest \
URLEncodeTest \
F16Test
F16Test \
A51Test
# ReportingTest
@ -72,7 +74,8 @@ noinst_HEADERS = \
URLEncode.h \
Utils.h \
Logger.h \
sqlite3util.h
sqlite3util.h \
A51.h
URLEncodeTest_SOURCES = URLEncodeTest.cpp
URLEncodeTest_LDADD = libcommon.la
@ -108,6 +111,9 @@ LogTest_LDADD = libcommon.la $(SQLITE_LA)
F16Test_SOURCES = F16Test.cpp
A51Test_SOURCES = A51Test.cpp
A51Test_LDADD = libcommon.la
MOSTLYCLEANFILES += testSource testDestination

86
Reporting.cpp
View File

@ -1,6 +1,6 @@
/**@file Module for performance-reporting mechanisms. */
/*
* Copyright 2012 Range Networks, Inc.
* Copyright 2012, 2013 Range Networks, Inc.
*
* This software is distributed under the terms of the GNU Affero Public License.
* See the COPYING file in the main directory for details.
@ -53,11 +53,25 @@ ReportingTable::ReportingTable(const char* filename)
if (!sqlite3_command(mDB,createReportingTable)) {
gLogEarly(LOG_EMERG | mFacility, "cannot create reporting table in database at %s, error message: %s", filename, sqlite3_errmsg(mDB));
}
// Set high-concurrency WAL mode.
if (!sqlite3_command(mDB,enableWAL)) {
gLogEarly(LOG_EMERG | mFacility, "Cannot enable WAL mode on database at %s, error message: %s", filename, sqlite3_errmsg(mDB));
}
// Start the commit thread
mBatchCommitter.start((void*(*)(void*))reportingBatchCommitter,NULL);
}
bool ReportingTable::create(const char* paramName)
{
// add this report name to the batch map
mLock.lock();
if (mBatch.find(paramName) == mBatch.end()) {
mBatch[paramName] = 0;
}
mLock.unlock();
// and to the database
char cmd[200];
sprintf(cmd,"INSERT OR IGNORE INTO REPORTING (NAME,CLEAREDTIME) VALUES (\"%s\",%ld)", paramName, time(NULL));
if (!sqlite3_command(mDB,cmd)) {
@ -71,12 +85,10 @@ bool ReportingTable::create(const char* paramName)
bool ReportingTable::incr(const char* paramName)
{
char cmd[200];
sprintf(cmd,"UPDATE REPORTING SET VALUE=VALUE+1, UPDATETIME=%ld WHERE NAME=\"%s\"", time(NULL), paramName);
if (!sqlite3_command(mDB,cmd)) {
gLogEarly(LOG_CRIT|mFacility, "cannot increment reporting parameter %s, error message: %s", paramName, sqlite3_errmsg(mDB));
return false;
}
mLock.lock();
mBatch[paramName]++;
mLock.unlock();
return true;
}
@ -106,6 +118,19 @@ bool ReportingTable::clear(const char* paramName)
}
bool ReportingTable::clear()
{
char cmd[200];
sprintf(cmd,"UPDATE REPORTING SET VALUE=0, UPDATETIME=0, CLEAREDTIME=%ld", time(NULL));
if (!sqlite3_command(mDB,cmd)) {
gLogEarly(LOG_CRIT|mFacility, "cannot clear reporting table, error message: %s", sqlite3_errmsg(mDB));
return false;
}
return true;
}
bool ReportingTable::create(const char* baseName, unsigned minIndex, unsigned maxIndex)
{
size_t sz = strlen(baseName);
@ -140,6 +165,53 @@ bool ReportingTable::clear(const char* baseName, unsigned index)
return clear(name);
}
bool ReportingTable::commit()
{
ReportBatch oustanding;
ReportBatch::iterator mp;
unsigned oustandingCount = 0;
// copy out to free up access to mBatch as quickly as possible
mLock.lock();
mp = mBatch.begin();
while (mp != mBatch.end()) {
if (mp->second > 0) {
oustanding[mp->first] = mp->second;
mBatch[mp->first] = 0;
oustandingCount++;
}
mp++;
}
mLock.unlock();
// now actually write them into the db if needed
if (oustandingCount > 0) {
Timeval timer;
char cmd[200];
// TODO : could wrap this in a BEGIN; COMMIT; pair to transactionize these X UPDATEs
mp = oustanding.begin();
while (mp != oustanding.end()) {
sprintf(cmd,"UPDATE REPORTING SET VALUE=VALUE+%u, UPDATETIME=%ld WHERE NAME=\"%s\"", mp->second, time(NULL), mp->first.c_str());
if (!sqlite3_command(mDB,cmd)) {
LOG(CRIT) << "could not increment reporting parameter " << mp->first << ", error message: " << sqlite3_errmsg(mDB);
}
mp++;
}
LOG(INFO) << "wrote " << oustandingCount << " entries in " << timer.elapsed() << "ms";
}
return true;
}
extern ReportingTable gReports;
void* reportingBatchCommitter(void*)
{
while (true) {
sleep(10);
gReports.commit();
}
return NULL;
}

18
Reporting.h
View File

@ -1,6 +1,6 @@
/**@file Module for performance-reporting mechanisms. */
/*
* Copyright 2012 Range Networks, Inc.
* Copyright 2012, 2013 Range Networks, Inc.
*
* This software is distributed under the terms of the GNU Affero Public License.
* See the COPYING file in the main directory for details.
@ -28,7 +28,12 @@
#include <sqlite3util.h>
#include <ostream>
#include <map>
#include <string>
#include <Threads.h>
#include <Timeval.h>
typedef std::map<std::string, unsigned> ReportBatch;
/**
Collect performance statistics into a database.
@ -40,6 +45,9 @@ class ReportingTable {
sqlite3* mDB; ///< database connection
int mFacility; ///< rsyslogd facility
ReportBatch mBatch; ///< batch of report updates, not yet stored in the db
mutable Mutex mLock; ///< control for multithreaded read/write access to the batch
Thread mBatchCommitter; ///< thread responsible for committing batches of report updates to the db
@ -69,6 +77,9 @@ class ReportingTable {
/** Take a max of an indexed parameter. */
bool max(const char* paramName, unsigned index, unsigned newVal);
/** Clear the whole table. */
bool clear();
/** Clear a value. */
bool clear(const char* paramName);
@ -78,8 +89,13 @@ class ReportingTable {
/** Dump the database to a stream. */
void dump(std::ostream&) const;
/** Commit outstanding report updates to the database */
bool commit();
};
/** Periodically triggers ReportingTable::commit(). */
void* reportingBatchCommitter(void*);
#endif

63
ReportingTest.cpp Normal file
View File

@ -0,0 +1,63 @@
/*
* Copyright 2012 Range Networks, Inc.
*
* This software is distributed under the terms of the GNU Affero Public License.
* See the COPYING file in the main directory for details.
*
* This use of this software may be subject to additional restrictions.
* See the LEGAL file in the main directory for details.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "stdlib.h"
#include <syslog.h>
#include <time.h>
#include "Reporting.h"
int main(int argc, char *argv[])
{
srandom(time(NULL));
ReportingTable rpt("./test.db",LOG_LOCAL7);
rpt.create("count1");
rpt.create("count2");
rpt.create("max1");
rpt.create("max2");
rpt.incr("count1");
rpt.incr("count2");
rpt.incr("count1");
rpt.clear("max1");
rpt.max("max1",random());
rpt.max("max2",random());
rpt.create("indexed",5,10);
for (int i=0; i<20; i++) {
rpt.incr("indexed",random()%6 + 5);
}
rpt.create("indexedMax",5,10);
for (int i=5; i<=10; i++) {
rpt.max("indexedMax", i, random());
}
}

2
Utils.cpp
View File

@ -193,7 +193,7 @@ std::ostream& operator<<(std::ostream& os, const Statistic<double> &stat) { stat
std::string replaceAll(const std::string input, const std::string search, const std::string replace)
{
std::string output = input;
int index = 0;
unsigned index = 0;
while (true) {
index = output.find(search, index);

4
example.config
View File

@ -12,7 +12,7 @@ key5 10 11 13 15
# This file is also used to test the logging system.
Log.Alarms.Max 10
Log.Alarms.Max 20
Log.Alarms.TargetIP 127.0.0.1
Log.Alarms.TargetPort 10101
Log.Level INFO
Log.Level NOTICE

68
sqlite3util.cpp
View File

@ -15,15 +15,21 @@
// Wrappers to sqlite operations.
// These will eventually get moved to commonlibs.
int sqlite3_prepare_statement(sqlite3* DB, sqlite3_stmt **stmt, const char* query)
const char* enableWAL = {
"PRAGMA journal_mode=WAL"
};
int sqlite3_prepare_statement(sqlite3* DB, sqlite3_stmt **stmt, const char* query, unsigned retries)
{
int src = SQLITE_BUSY;
while (src==SQLITE_BUSY) {
src = sqlite3_prepare_v2(DB,query,strlen(query),stmt,NULL);
if (src==SQLITE_BUSY) {
usleep(100000);
}
}
for (unsigned i = 0; i < retries; i++) {
src = sqlite3_prepare_v2(DB,query,strlen(query),stmt,NULL);
if (src != SQLITE_BUSY && src != SQLITE_LOCKED) {
break;
}
usleep(200);
}
if (src) {
fprintf(stderr,"sqlite3_prepare_v2 failed for \"%s\": %s\n",query,sqlite3_errmsg(DB));
sqlite3_finalize(*stmt);
@ -31,15 +37,17 @@ int sqlite3_prepare_statement(sqlite3* DB, sqlite3_stmt **stmt, const char* quer
return src;
}
int sqlite3_run_query(sqlite3* DB, sqlite3_stmt *stmt)
int sqlite3_run_query(sqlite3* DB, sqlite3_stmt *stmt, unsigned retries)
{
int src = SQLITE_BUSY;
while (src==SQLITE_BUSY) {
src = sqlite3_step(stmt);
if (src==SQLITE_BUSY) {
usleep(100000);
}
}
for (unsigned i = 0; i < retries; i++) {
src = sqlite3_step(stmt);
if (src != SQLITE_BUSY && src != SQLITE_LOCKED) {
break;
}
usleep(200);
}
if ((src!=SQLITE_DONE) && (src!=SQLITE_ROW)) {
fprintf(stderr,"sqlite3_run_query failed: %s: %s\n", sqlite3_sql(stmt), sqlite3_errmsg(DB));
}
@ -48,16 +56,16 @@ int sqlite3_run_query(sqlite3* DB, sqlite3_stmt *stmt)
bool sqlite3_exists(sqlite3* DB, const char *tableName,
const char* keyName, const char* keyData)
const char* keyName, const char* keyData, unsigned retries)
{
size_t stringSize = 100 + strlen(tableName) + strlen(keyName) + strlen(keyData);
char query[stringSize];
sprintf(query,"SELECT * FROM %s WHERE %s == \"%s\"",tableName,keyName,keyData);
// Prepare the statement.
sqlite3_stmt *stmt;
if (sqlite3_prepare_statement(DB,&stmt,query)) return false;
if (sqlite3_prepare_statement(DB,&stmt,query,retries)) return false;
// Read the result.
int src = sqlite3_run_query(DB,stmt);
int src = sqlite3_run_query(DB,stmt,retries);
sqlite3_finalize(stmt);
// Anything there?
return (src == SQLITE_ROW);
@ -67,16 +75,16 @@ bool sqlite3_exists(sqlite3* DB, const char *tableName,
bool sqlite3_single_lookup(sqlite3* DB, const char *tableName,
const char* keyName, const char* keyData,
const char* valueName, unsigned &valueData)
const char* valueName, unsigned &valueData, unsigned retries)
{
size_t stringSize = 100 + strlen(valueName) + strlen(tableName) + strlen(keyName) + strlen(keyData);
char query[stringSize];
sprintf(query,"SELECT %s FROM %s WHERE %s == \"%s\"",valueName,tableName,keyName,keyData);
// Prepare the statement.
sqlite3_stmt *stmt;
if (sqlite3_prepare_statement(DB,&stmt,query)) return false;
if (sqlite3_prepare_statement(DB,&stmt,query,retries)) return false;
// Read the result.
int src = sqlite3_run_query(DB,stmt);
int src = sqlite3_run_query(DB,stmt,retries);
bool retVal = false;
if (src == SQLITE_ROW) {
valueData = (unsigned)sqlite3_column_int64(stmt,0);
@ -90,7 +98,7 @@ bool sqlite3_single_lookup(sqlite3* DB, const char *tableName,
// This function returns an allocated string that must be free'd by the caller.
bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
const char* keyName, const char* keyData,
const char* valueName, char* &valueData)
const char* valueName, char* &valueData, unsigned retries)
{
valueData=NULL;
size_t stringSize = 100 + strlen(valueName) + strlen(tableName) + strlen(keyName) + strlen(keyData);
@ -98,9 +106,9 @@ bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
sprintf(query,"SELECT %s FROM %s WHERE %s == \"%s\"",valueName,tableName,keyName,keyData);
// Prepare the statement.
sqlite3_stmt *stmt;
if (sqlite3_prepare_statement(DB,&stmt,query)) return false;
if (sqlite3_prepare_statement(DB,&stmt,query,retries)) return false;
// Read the result.
int src = sqlite3_run_query(DB,stmt);
int src = sqlite3_run_query(DB,stmt,retries);
bool retVal = false;
if (src == SQLITE_ROW) {
const char* ptr = (const char*)sqlite3_column_text(stmt,0);
@ -115,7 +123,7 @@ bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
// This function returns an allocated string that must be free'd by tha caller.
bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
const char* keyName, unsigned keyData,
const char* valueName, char* &valueData)
const char* valueName, char* &valueData, unsigned retries)
{
valueData=NULL;
size_t stringSize = 100 + strlen(valueName) + strlen(tableName) + strlen(keyName) + 20;
@ -123,9 +131,9 @@ bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
sprintf(query,"SELECT %s FROM %s WHERE %s == %u",valueName,tableName,keyName,keyData);
// Prepare the statement.
sqlite3_stmt *stmt;
if (sqlite3_prepare_statement(DB,&stmt,query)) return false;
if (sqlite3_prepare_statement(DB,&stmt,query,retries)) return false;
// Read the result.
int src = sqlite3_run_query(DB,stmt);
int src = sqlite3_run_query(DB,stmt,retries);
bool retVal = false;
if (src == SQLITE_ROW) {
const char* ptr = (const char*)sqlite3_column_text(stmt,0);
@ -139,15 +147,15 @@ bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
bool sqlite3_command(sqlite3* DB, const char* query)
bool sqlite3_command(sqlite3* DB, const char* query, unsigned retries)
{
// Prepare the statement.
sqlite3_stmt *stmt;
if (sqlite3_prepare_statement(DB,&stmt,query)) return false;
if (sqlite3_prepare_statement(DB,&stmt,query,retries)) return false;
// Run the query.
int src = sqlite3_run_query(DB,stmt);
int src = sqlite3_run_query(DB,stmt,retries);
sqlite3_finalize(stmt);
return src==SQLITE_DONE;
return (src==SQLITE_DONE || src==SQLITE_OK || src==SQLITE_ROW);
}

20
sqlite3util.h
View File

@ -3,27 +3,33 @@
#include <sqlite3.h>
int sqlite3_prepare_statement(sqlite3* DB, sqlite3_stmt **stmt, const char* query);
// (pat) Dont put statics in .h files - they generate a zillion g++ error messages.
extern const char *enableWAL;
//static const char* enableWAL = {
// "PRAGMA journal_mode=WAL"
//};
int sqlite3_run_query(sqlite3* DB, sqlite3_stmt *stmt);
int sqlite3_prepare_statement(sqlite3* DB, sqlite3_stmt **stmt, const char* query, unsigned retries = 5);
int sqlite3_run_query(sqlite3* DB, sqlite3_stmt *stmt, unsigned retries = 5);
bool sqlite3_single_lookup(sqlite3* DB, const char *tableName,
const char* keyName, const char* keyData,
const char* valueName, unsigned &valueData);
const char* valueName, unsigned &valueData, unsigned retries = 5);
bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
const char* keyName, const char* keyData,
const char* valueName, char* &valueData);
const char* valueName, char* &valueData, unsigned retries = 5);
// This function returns an allocated string that must be free'd by the caller.
bool sqlite3_single_lookup(sqlite3* DB, const char* tableName,
const char* keyName, unsigned keyData,
const char* valueName, char* &valueData);
const char* valueName, char* &valueData, unsigned retries = 5);
bool sqlite3_exists(sqlite3* DB, const char* tableName,
const char* keyName, const char* keyData);
const char* keyName, const char* keyData, unsigned retries = 5);
/** Run a query, ignoring the result; return true on success. */
bool sqlite3_command(sqlite3* DB, const char* query);
bool sqlite3_command(sqlite3* DB, const char* query, unsigned retries = 5);
#endif