Skip to content

Commit 37cbd13

Browse files
author
Worvast
authored
Merge pull request #24 from Worvast/master
New version 3
2 parents cbe0d94 + b93ede6 commit 37cbd13

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+1327
-471
lines changed

CHANGELOG.md

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,42 @@ All notable changes to this project will be documented in this file.
44
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
55
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
66

7+
## [3.0.0] - 2020-04-28
8+
#### Added
9+
* Faker
10+
* Faker time periods for probability and frequency
11+
* SimulationGenerator
12+
* New providers
13+
* New and splitted documentation of use
14+
* Examples for scripts and CLI shell
15+
* Generators/Faker now accept custom providers, custom functions
16+
from base python or custom scripts for use in templates
17+
* CLI Shell has now all necessary flags for use all generators
18+
19+
#### Changed
20+
* General
21+
* Modified base devo-sender from version 3.0.x to 3.3.0
22+
* Update dependencies to a new majors versions
23+
* Set dependencies to fixed versions (more maintenance, but much more security and reliability)
24+
* Sorter
25+
* file_sorted_join moved from devoutils.sorter to devoutils.fileio
26+
* Remove Python 2 compatibility
27+
* Faker
28+
* Faker cli add verbose mode to show the events in the console
29+
* Add file_name param to define a file to store events in batch mode or
30+
for testing, store in a file but do not send it.
31+
* Generator template are moved for Template
32+
* Generator date_generator are moved for Template
33+
* Generators name changed, example: BatchSender for BatchFakerGenerator
34+
* Now you can make functions callable in each line of Faker Jinja2 Template
35+
* null_date_generator (Default date generator) isn't a generator now, its a normal function
36+
* Template.process() now not create new DateGenerator in each call
37+
* Numbers_providers has PEP8 variable names
38+
* `freq` var/flag in all code are now `frequency`
39+
* `prob` var/flag in all code are now `probability`
40+
* CLI Shell --config file now require all vars in "faker" object, and all vars for Sender in "sender" object
41+
42+
743
## [2.0.0] - 2019-07-02
844
#### Changed
945
* Modified base devo-sender from version 2.x to >=3.0.1
@@ -28,4 +64,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
2864

2965
## [1.0.0] - 2019-01-16
3066
#### Added
31-
* On the ninth month of the year, a Devo programmer finally decided to publish a part of the Utils and rest in peace.
67+
* On the ninth month of the year, a Devo programmer finally decided to publish a part of the Utils and rest in peace.

README.md

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[![relese-next Build Status](https://travis-ci.com/DevoInc/python-utils.svg?branch=master)](https://travis-ci.com/DevoInc/python-utils) [![LICENSE](https://img.shields.io/dub/l/vibe-d.svg)](https://github.com/DevoInc/python-utils/blob/master/LICENSE)
22

3-
[![wheel](https://img.shields.io/badge/wheel-yes-brightgreen.svg)](https://pypi.org/project/devo-utils/) [![version](https://img.shields.io/badge/version-2.0.1-blue.svg)](https://pypi.org/project/devo-utils/) [![python](https://img.shields.io/badge/python-3.5%20%7C%203.6%20%7C%203.7-blue.svg)](https://pypi.org/project/devo-utils/)
3+
[![wheel](https://img.shields.io/badge/wheel-yes-brightgreen.svg)](https://pypi.org/project/devo-utils/) [![version](https://img.shields.io/badge/version-3.0.0-blue.svg)](https://pypi.org/project/devo-utils/) [![python](https://img.shields.io/badge/python-3.5%20%7C%203.6%20%7C%203.7%20%7C%203.8-blue.svg)](https://pypi.org/project/devo-utils/)
44

55

66
# Devo Python Utils
@@ -41,8 +41,18 @@ You can use sources files, clonning the project too:
4141

4242
You have specific documentation in _[docs](docs)_ folder for each part of SDK:
4343
* [Faker: fake data](docs/faker.md)
44-
* [File IO](docs/io/fileio.md)
45-
* [Sorting data](docs/sorter.md)
44+
* [File IO](docs/fileio.md)
45+
* [Sorting data](docs/sorter.md)If you wrap a line with the block creator
46+
47+
{% - set .... -%}
48+
49+
Faker will not treat that line as a send line, if a variable is created without the hyphens, for example:
50+
51+
{% set ....%}
52+
53+
Faker will believe that it is a line to send, so you have to keep this in mind
54+
55+
And examples in `examples` folder.
4656

4757

4858
## Contributing

devoutils/__version__.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
__description__ = 'Devo Python Library.'
22
__url__ = 'http://www.devo.com'
3-
__version__ = "2.0.1"
3+
__version__ = "3.0.0"
44
__author__ = 'Devo'
55
__author_email__ = 'support@devo.com'
66
__license__ = 'MIT'
7-
__copyright__ = 'Copyright 2019 Devo'
7+
__copyright__ = 'Copyright 2020 Devo'

devoutils/faker/__init__.py

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1-
from .senders.batch_sender import BatchSender
2-
from .senders.file_sender import FileSender
3-
from .senders.syslog_sender import SyslogSender
4-
from .senders.syslog_raw_sender import SyslogRawSender
5-
from .senders.template_parser import TemplateParser
1+
from .generators.batch_fake_generator import BatchFakeGenerator
2+
from .generators.file_fake_generator import FileFakeGenerator
3+
from .generators.syslog_fake_generator import SyslogFakeGenerator
4+
from .generators.syslog_raw_fake_generator import SyslogRawFakeGenerator
5+
from .generators.template_parser import TemplateParser
6+
from .generators.simulation_fake_generator import SimulationFakeGenerator
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from .batch_fake_generator import BatchFakeGenerator
2+
from .file_fake_generator import FileFakeGenerator
3+
from .syslog_fake_generator import SyslogFakeGenerator
4+
from .syslog_raw_fake_generator import SyslogRawFakeGenerator
5+
from .template_parser import TemplateParser
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# -*- coding: utf-8 -*-
2+
"""Base Sender, father of all other providers"""
3+
import random
4+
import threading
5+
import time
6+
from datetime import datetime, timedelta
7+
from pycron import is_now
8+
from .template_parser import TemplateParser
9+
10+
11+
class BaseFakeGenerator(threading.Thread):
12+
"""Base provider main class"""
13+
14+
def __init__(self, engine=None, template=None, **kwargs):
15+
"""
16+
Init BaseFakeGenerator, all generator use this class as father
17+
18+
:param engine (Sender): Sender object, if neccesary, for send data
19+
:param template (str, bytes): Template, read, in a variable, not the
20+
path to a file
21+
:param kwargs: List of accepted variables below
22+
:param time_rules (dict): Time rules as crondates format for
23+
variable frequeny and probability
24+
:param frequency (list): Fixed shipping frequency list (min, max)
25+
in seconds
26+
:param probability (int): Fixed shipping probability
27+
from 0 to 100(%)
28+
:param interactive (bool): Interactive mode
29+
:param simulation (bool): Simulation mode, not send/write if true
30+
:param verbose (bool): Verbose mode
31+
:param date_generator (object): Alternative date generator
32+
for templates
33+
:param providers (dict): Custom providers dict
34+
"""
35+
36+
threading.Thread.__init__(self)
37+
self.engine = engine
38+
39+
if kwargs.get('time_rules', None) is not None:
40+
self.time_rules = kwargs.get("time_rules")
41+
self.set_last_time_rule()
42+
43+
self.cron_child = threading.Thread(
44+
target=self.check_time_rules
45+
)
46+
self.cron_child.setDaemon(True)
47+
self.cron_child.start()
48+
else:
49+
self.prob = kwargs.get('probability', 100)
50+
self.freq = kwargs.get('frequency', (1, 1))
51+
52+
self.interactive = kwargs.get('interactive', False)
53+
self.simulation = kwargs.get('simulation', False)
54+
self.verbose = kwargs.get('verbose', False)
55+
self.parser = TemplateParser(template=str(template),
56+
providers=kwargs.get('providers', {}),
57+
date_generator=
58+
kwargs.get('date_generator', None))
59+
60+
def process(self, date_generator=None, **kwargs):
61+
"""Process template"""
62+
return self.parser.process(date_generator=date_generator,
63+
**kwargs)
64+
65+
def wait(self):
66+
"""Time to wait between events"""
67+
# freq[0] is the minimum
68+
# freq[1] is the maximum
69+
if self.freq[0] == self.freq[1]:
70+
secs = self.freq[0]
71+
elif self.freq[1] < self.freq[0]:
72+
secs = random.uniform(self.freq[1], self.freq[0])
73+
else:
74+
secs = random.uniform(self.freq[0], self.freq[1])
75+
time.sleep(secs)
76+
77+
def set_last_time_rule(self):
78+
"""
79+
When crondate time rules are used, it is found which one should have
80+
been executed last and the probability and frequency are set
81+
:return None
82+
"""
83+
now = datetime.now()
84+
try:
85+
for _ in range(0, 525600):
86+
for item in self.time_rules:
87+
if is_now(item["rule"], now):
88+
self.prob = item['probability']
89+
self.freq = item['frequency']
90+
raise Exception('found')
91+
now = now - timedelta(minutes=1)
92+
except Exception as inst:
93+
if inst.args[0] != "found":
94+
raise Exception(inst)
95+
96+
def check_time_rules(self):
97+
"""
98+
Controls every minute if a change in frequency and probability
99+
is to be made based on the specified rules
100+
:return:
101+
"""
102+
while True:
103+
for item in self.time_rules:
104+
if is_now(item["rule"]):
105+
self.prob = item['probability']
106+
self.freq = item['frequency']
107+
break
108+
time.sleep(60)
109+
110+
def probability(self):
111+
"""Calculate probability"""
112+
k = random.randint(0, 100)
113+
if k <= int(self.prob):
114+
return True
115+
return False
116+
117+
def run(self):
118+
"""Run example (for override)"""
119+
while True:
120+
if self.probability():
121+
# Do something
122+
pass
123+
self.wait()
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
"""Batch sender provider"""
2+
import datetime
3+
import os.path
4+
import random
5+
import sys
6+
import click
7+
8+
from .base_fake_generator import BaseFakeGenerator
9+
10+
11+
class BatchFakeGenerator(BaseFakeGenerator):
12+
"""
13+
Generates a lot of events in a single batch without any waits, it
14+
supports generating events from the past
15+
"""
16+
17+
GENERATION_ENDED_TOKEN = '###END###'
18+
19+
def __init__(self, template=None, start_date=None, end_date=None,
20+
**kwargs):
21+
22+
BaseFakeGenerator.__init__(self, template=template, **kwargs)
23+
24+
self._start_date = start_date
25+
self._end_date = end_date
26+
self.date_format = kwargs.get('date_format', "%Y-%m-%d %H:%M:%S.%f")
27+
self.dont_remove_microseconds = kwargs.get('dont_remove_microseconds',
28+
False)
29+
self.file_name = kwargs.get("file_name") if kwargs.get("file_name") \
30+
else "eventbatch.log"
31+
32+
@staticmethod
33+
def date_range(start_date=None, end_date=None, frequency=None,
34+
probability=None,
35+
date_format="%Y-%m-%d %H:%M:%S.%f",
36+
dont_remove_microseconds=True):
37+
"""
38+
Generates date range
39+
:param start_date:me objects between a start date and an end date with
40+
some given frequency,
41+
Some events can be piled up in the same millisecond, this happens
42+
sometimes with real data.
43+
:param end_date:
44+
:param start_date:
45+
:param frequency:
46+
:param probability:
47+
:param date_format:
48+
:param dont_remove_microseconds:
49+
:return:
50+
"""
51+
52+
# Control how fast events are spaced between them with a
53+
# frequency (--frequency argument)
54+
millis = 0
55+
idx = 0
56+
while True:
57+
58+
millis_increment = random.uniform(frequency[0],
59+
frequency[1]) * 1000
60+
61+
# Add some randomness to the event generation with
62+
# the --probability argument
63+
k = random.randint(0, 100)
64+
if k <= int(probability):
65+
millis += millis_increment
66+
67+
idx += 1
68+
next_date = start_date + datetime.timedelta(milliseconds=millis)
69+
70+
if next_date > end_date:
71+
yield BatchFakeGenerator.GENERATION_ENDED_TOKEN
72+
else:
73+
if dont_remove_microseconds:
74+
yield next_date.strftime(date_format)
75+
else:
76+
yield next_date.strftime(date_format)[:-3]
77+
78+
def run(self):
79+
"""Run function for cli or call function"""
80+
if os.path.exists(self.file_name):
81+
raise ValueError('an {} file already exists, remove it before'
82+
' generating new events'.format(self.file_name))
83+
84+
counter = 0
85+
date_generator = self.date_range(
86+
self._start_date, self._end_date, self.freq, self.prob,
87+
self.date_format, self.dont_remove_microseconds)
88+
with open(self.file_name, "w") as f:
89+
while True:
90+
lines = self.process(date_generator=date_generator).split('\n')
91+
for line in lines:
92+
if BatchFakeGenerator.GENERATION_ENDED_TOKEN in line:
93+
print("Wrote {} events in {}".format(counter,
94+
self.file_name),
95+
file=sys.stdout)
96+
return
97+
if counter % 100 == 0:
98+
click.echo('Wrote {} lines'.format(counter),
99+
file=sys.stderr)
100+
click.echo(line[:40], file=sys.stderr)
101+
if self.verbose:
102+
print(line, file=sys.stdout)
103+
f.write(line+"\n")
104+
counter += 1
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# -*- coding: utf-8 -*-
2+
"""File Sender provider"""
3+
from datetime import datetime
4+
from .realtime_fake_generator import RealtimeFakeGenerator
5+
6+
7+
class FileFakeGenerator(RealtimeFakeGenerator):
8+
"""Generate a lot of events from file"""
9+
def __init__(self, template=None, **kwargs):
10+
RealtimeFakeGenerator.__init__(self, template=template, **kwargs)
11+
self.last_flush = int(datetime.now().timestamp())
12+
self.file_name = kwargs.get('file_name', "safestream.log")
13+
self.file = None
14+
15+
def write_row(self, message=None):
16+
"""Write row to file"""
17+
self.file.write(message)
18+
self.file.write('\n')
19+
now = int(datetime.now().timestamp())
20+
# flush every 5 secs
21+
if now - self.last_flush > 5:
22+
self.last_flush = now
23+
self.file.flush()
24+
25+
def run(self):
26+
"""Run function for cli or call function"""
27+
with open(self.file_name, "a") as self.file:
28+
self.realtime_iteration(write_function=self.write_row)
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# -*- coding: utf-8 -*-
2+
"""realtime base sender provider"""
3+
from datetime import datetime
4+
from .base_fake_generator import BaseFakeGenerator
5+
6+
7+
class RealtimeFakeGenerator(BaseFakeGenerator):
8+
"""Realtime fake data generation"""
9+
def __init__(self, engine=None, template=None, **kwargs):
10+
"""Realtime fake data generation"""
11+
BaseFakeGenerator.__init__(self, engine=engine, template=template,
12+
**kwargs)
13+
14+
def realtime_iteration(self, write_function=None):
15+
"""Realtime function for parse and send/show generated data"""
16+
while True:
17+
lines = self.process().split('\n')
18+
for line in lines:
19+
if self.probability():
20+
if not self.simulation:
21+
write_function(line)
22+
now = datetime.utcnow().ctime()
23+
if self.verbose:
24+
print('{0} => {1}'.format(now, str(line)))
25+
else:
26+
now = datetime.utcnow().ctime()
27+
if self.verbose:
28+
print('{0} => Skipped by prob.'.format(now))
29+
30+
if self.interactive:
31+
input("» Press Enter for next iteration «")
32+
else:
33+
self.wait()

0 commit comments

Comments
 (0)